diff --git a/.gitignore b/.gitignore
index 1cd70d1f9..04ea46037 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
 *.pyc
 .DS_Store
 **.log
+pretrained/
 .pytest*
 .venv*
 .coverage
diff --git a/README.md b/README.md
index d417c1dc6..b1bef9ef4 100644
--- a/README.md
+++ b/README.md
@@ -1,14 +1,16 @@
-# Model Zoo for Intel® Architecture
+# Intel® AI Reference Models
 
 This repository contains **links to pre-trained models, sample scripts, best practices, and step-by-step tutorials** for many popular open-source machine learning models optimized by Intel to run on Intel® Xeon® Scalable processors and Intel® Data Center GPUs.
 
-Model packages and containers for running the Model Zoo's workloads can be found at the [Intel® Developer Catalog](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html).
+Containers for running the workloads can be found at the [Intel® Developer Catalog](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html).
 
-## Purpose of the Model Zoo
+[Intel® AI Reference Models in a Jupyter Notebook](/notebooks/README.md) is also available for the [listed workloads](/notebooks/README.md#supported-models)
 
-  - Demonstrate the AI workloads and deep learning models Intel has optimized and validated to run on Intel hardware
-  - Show how to efficiently execute, train, and deploy Intel-optimized models
-  - Make it easy to get started running Intel-optimized models on Intel hardware in the cloud or on bare metal
+## Purpose of Intel® AI Reference Models
+
+Intel optimizes popular deep learning frameworks such as TensorFlow* and PyTorch* by contributing to the upstream projects. Additional optimizations are built into plugins/extensions such as the [Intel Extension for Pytorch*](https://github.com/intel/intel-extension-for-pytorch) and the [Intel Extension for TensorFlow*](https://github.com/intel/intel-extension-for-tensorflow). Popular neural network models running against common datasets are the target workloads that drive these optimizations. 
+
+The purpose of the Intel® AI Reference Models repository (and associated containers) is to quickly replicate the complete software environment that demonstrates the best-known performance of each of these target model/dataset combinations. When executed in optimally-configured hardware environments, these software environments showcase the AI capabilities of Intel platforms. 
 
 ***DISCLAIMER: These scripts are not intended for benchmarking Intel platforms.
 For any performance and/or benchmarking information on specific Intel platforms, visit [https://www.intel.ai/blog](https://www.intel.ai/blog).***
@@ -16,12 +18,12 @@ For any performance and/or benchmarking information on specific Intel platforms,
 Intel is committed to the respect of human rights and avoiding complicity in human rights abuses, a policy reflected in the [Intel Global Human Rights Principles](https://www.intel.com/content/www/us/en/policy/policy-human-rights.html). Accordingly, by accessing the Intel material on this platform you agree that you will not use the material in a product or application that causes or contributes to a violation of an internationally recognized human right.
 
 ## License
-The Model Zoo for Intel® Architecture is licensed under [Apache License Version 2.0](https://github.com/IntelAI/models/blob/master/LICENSE).
+The Intel® AI Reference Models is licensed under [Apache License Version 2.0](https://github.com/intel/ai-reference-models/blob/master/LICENSE).
 
 ## Datasets
 To the extent that any public datasets are referenced by Intel or accessed using tools or code on this site those datasets are provided by the third party indicated as the data source. Intel does not create the data, or datasets, and does not warrant their accuracy or quality. By accessing the public dataset(s) you agree to the terms associated with those datasets and that your use complies with the applicable license.
 
-Please check the list of datasets used in Model Zoo for Intel® Architecture in [datasets directory](/datasets).
+Please check the list of datasets used in Intel® AI Reference Models in [datasets directory](/datasets).
 
 Intel expressly disclaims the accuracy, adequacy, or completeness of any public datasets, and is not liable for any errors, omissions, or defects in the data, or for any reliance on the data. Intel is not liable for any liability or damages relating to your use of public datasets.
 
@@ -30,7 +32,7 @@ The model documentation in the tables below have information on the
 prerequisites to run each model. The model scripts run on Linux. Certain
 models are also able to run using bare metal on Windows. For more information
 and a list of models that are supported on Windows, see the
-[documentation here](/docs/general/Windows.md#using-intel-model-zoo-on-windows-systems).
+[documentation here](/docs/general/Windows.md#using-intel-ai-reference-models-on-windows-systems).
 
 Instructions available to run on [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx).
 
@@ -73,7 +75,6 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 
 | Model                                                    | Framework  | Mode      | Model Documentation | Benchmark/Test Dataset |
 | -------------------------------------------------------- | ---------- | ----------| ------------------- | ---------------------- |
-| [3D U-Net](https://arxiv.org/pdf/1606.06650.pdf)         | TensorFlow | Inference | [FP32](/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/README.md) | [BRATS 2018](https://github.com/IntelAI/models/tree/master/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32#datasets) |
 | [3D U-Net MLPerf*](https://arxiv.org/pdf/1606.06650.pdf) | TensorFlow | Inference | [FP32 BFloat16 Int8](/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md) | [BRATS 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) |
 | [3D U-Net MLPerf*](https://arxiv.org/pdf/1606.06650.pdf) [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | Tensorflow | Inference | [FP32 BFloat16 Int8 BFloat32](/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_SPR_Baremetal.md) | [BRATS 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) |
 | [MaskRCNN](https://arxiv.org/abs/1703.06870)             | TensorFlow | Inference | [FP32](/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md) | [MS COCO 2014](https://github.com/IntelAI/models/tree/master/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32#datasets-and-pretrained-model) |
@@ -114,7 +115,6 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 
 | Model                                                 | Framework  | Mode      | Model Documentation | Benchmark/Test Dataset |
 | ----------------------------------------------------- | ---------- | ----------| ------------------- | ---------------------- |
-| [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf)  | TensorFlow | Inference | [Int8](/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md) [FP32](/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [R-FCN](https://arxiv.org/pdf/1605.06409.pdf)         | TensorFlow | Inference | [Int8 FP32](/benchmarks/object_detection/tensorflow/rfcn/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)| TensorFlow | Inference | [Int8 FP32 BFloat16](/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)  [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_SPR_baremetal.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
@@ -145,6 +145,9 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | [Wide & Deep Large Dataset](https://arxiv.org/pdf/1606.07792.pdf) | TensorFlow | Training | [FP32](/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md) | [Large Kaggle Display Advertising Challenge dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32#dataset) |
 | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md#datasets) |
 | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Training  | [FP32 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md#datasets) |
+| [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Inference | [FP32 FP16 BFloat16 BFloat32 Int8](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md) | [Criteo 1TB Click Logs dataset](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu#datasets) |
+| [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Training  | [FP32 FP16 BFloat16 BFloat32](/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md) | [Random dataset](/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu#datasets) |
+| [MEMREC-DLRM](https://arxiv.org/pdf/2305.07205.pdf)         | PyTorch | Inference  | [FP32](/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md#datasets) |
 
 ### Text-to-Speech
 
@@ -189,6 +192,8 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Training  | Max Series | [BFloat16](/quickstart/language_modeling/pytorch/bert_large/training/gpu/README.md) |
 |[BERT large](https://arxiv.org/pdf/1810.04805.pdf) | TensorFlow | Inference | Max Series | [FP32 FP16](/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md) |
 | [BERT large](https://arxiv.org/pdf/1810.04805.pdf) | TensorFlow | Training | Max Series | [BFloat16](/quickstart/language_modeling/tensorflow/bert_large/training/gpu/README.md) |
+| [DLRM](https://arxiv.org/pdf/1906.00091.pdf) | TensorFlow | Inference | Max Series | [FP16](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/README.md) |
+| [DLRM](https://arxiv.org/pdf/1906.00091.pdf) | TensorFlow | Training | Max Series | [BFloat16](/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/README.md) |
 
 ## How to Contribute
 If you would like to add a new benchmarking script, please use [this guide](/Contribute.md).
diff --git a/benchmarks/README.md b/benchmarks/README.md
deleted file mode 100644
index 51b603deb..000000000
--- a/benchmarks/README.md
+++ /dev/null
@@ -1,105 +0,0 @@
-# Model Zoo Scripts
-
-Training and inference scripts with TensorFlow optimizations that use the
-Intel® oneAPI Deep Neural Network Library (Intel® oneDNN) and
-Intel® Extension for PyTorch.
-
-## Prerequisites
-
-The model documentation in the tables below have information on the
-prerequisites to run each model. The model scripts run on Linux. Certain
-models are also able to run using bare metal on Windows. For more information
-and a list of models that are supported on Windows, see the
-[documentation here](/docs/general/Windows.md#using-intel-model-zoo-on-windows-systems).
-
-For information on running more advanced use cases using the workload containers see the:
-[advanced options documentation](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md).
-
-## TensorFlow Use Cases
-
-| Use Case                | Model              | Mode      | oneContainer Portal | Model Documentation | Dataset |
-| ------------------------| ------------------ | --------- | ------------------- | ------------------- | ------- |
-| Image Recognition       | [DenseNet169](https://arxiv.org/pdf/1608.06993.pdf) | Inference |  | [FP32](image_recognition/tensorflow/densenet169/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [Inception V3](https://arxiv.org/pdf/1512.00567.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv3-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv3-fp32-inference-tensorflow-container.html)| [Int8 FP32](image_recognition/tensorflow/inceptionv3/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [Inception V4](https://arxiv.org/pdf/1602.07261.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv4-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv4-fp32-inference-tensorflow-container.html) | [Int8 FP32](image_recognition/tensorflow/inceptionv4/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [MobileNet V1*](https://arxiv.org/pdf/1704.04861.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/mobilenetv1-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/mobilenetv1-fp32-inference-tensorflow-container.html) | [Int8 FP32 BFloat16](image_recognition/tensorflow/mobilenet_v1/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [MobileNet V2](https://arxiv.org/pdf/1801.04381.pdf) | Inference |  | [Int8 FP32 BFloat16](image_recognition/tensorflow/mobilenet_v2/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [ResNet 101](https://arxiv.org/pdf/1512.03385.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet101-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet101-fp32-inference-tensorflow-container.html) | [Int8 FP32](image_recognition/tensorflow/resnet101/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50-fp32-inference-tensorflow-container.html) | [Int8 FP32](image_recognition/tensorflow/resnet50/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50v1-5-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50v1-5-fp32-inference-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50v1-5-bfloat16-inference-tensorflow-container.html) | [Int8 FP32 BFloat16](image_recognition/tensorflow/resnet50v1_5/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | Training | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50v1-5-fp32-training-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/resnet50v1-5-bfloat16-training-tensorflow-container.html) | [FP32 BFloat16](image_recognition/tensorflow/resnet50v1_5/training/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [Vision Transformer](https://arxiv.org/abs/2010.11929) | Inference | Model Containers: | [FP32 BFloat16 FP16](https://github.com/IntelAI/models/benchmarks/image_recognition/tensorflow/vision_transformer/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Recognition       | [Vision Transformer](https://arxiv.org/abs/2010.11929) | Training | Model Containers: | [FP32 BFloat16 FP16](https://github.com/IntelAI/models/benchmarks/image_recognition/tensorflow/vision_transformer/training/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| Image Segmentation      | [3D U-Net](https://arxiv.org/pdf/1606.06650.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/3d-unet-fp32-inference-tensorflow-container.html) | [FP32](image_segmentation/tensorflow/3d_unet/inference/fp32/README.md) | [BRATS 2018](https://github.com/IntelAI/models/tree/master/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32#datasets) |
-| Image Segmentation      | [3D U-Net MLPerf*](https://arxiv.org/pdf/1606.06650.pdf) | Inference |  | [FP32 BFloat16 Int8](image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md) | [BRATS 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) |
-| Image Segmentation      | [MaskRCNN](https://arxiv.org/abs/1703.06870) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/mask-rcnn-fp32-inference-tensorflow-container.html) | [FP32](image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md) | [MS COCO 2014](https://github.com/IntelAI/models/tree/master/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32#datasets-and-pretrained-model) |
-| Image Segmentation      | [UNet](https://arxiv.org/pdf/1606.06650.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/unet-fp32-inference-tensorflow-container.html) | [FP32](image_segmentation/tensorflow/unet/inference/fp32/README.md) |
-| Language Modeling       | [BERT](https://arxiv.org/pdf/1810.04805.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/bert-large-fp32-inference-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/bert-large-bfloat16-inference-tensorflow-container.html) | [Int8](language_modeling/tensorflow/bert_large/inference/int8/README.md) [FP32](language_modeling/tensorflow/bert_large/inference/fp32/README.md) [BFloat16](language_modeling/tensorflow/bert_large/inference/bfloat16/README.md) | [SQuAD](https://github.com/IntelAI/models/tree/master/datasets/bert_data/README.md#inference) |
-| Language Modeling       | [BERT](https://arxiv.org/pdf/1810.04805.pdf) | Training | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/bert-large-fp32-training-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/bert-large-bfloat16-training-tensorflow-container.html) | [FP32](language_modeling/tensorflow/bert_large/training/fp32/Advanced.md) [BFloat16](language_modeling/tensorflow/bert_large/training/bfloat16/Advanced.md) [FP16](language_modeling/tensorflow/bert_large/training/fp16/Advanced.md) | [SQuAD](https://github.com/IntelAI/models/tree/master/datasets/bert_data/README.md#fine-tuning-with-bert-using-squad-data) and [MRPC](https://github.com/IntelAI/models/tree/master/datasets/bert_data/README.md#classification-training-with-bert) |
-| Language Modeling | [distilBERT](https://arxiv.org/abs/1910.01108) | Inference | Model Containers: | [FP32 BFloat16](https://github.com/IntelAI/models/benchmarks/language_modeling/tensorflow/distilbert_base/inference/README.md) | [SST-2](https://huggingface.co/datasets/sst2) |
-| Language Translation    | [BERT](https://arxiv.org/pdf/1810.04805.pdf) | Inference |  | [FP32](language_translation/tensorflow/bert/inference/README.md) | [MRPC](https://github.com/IntelAI/models/tree/master/datasets/bert_data/README.md#classification-training-with-bert) |
-| Language Translation    | [GNMT*](https://arxiv.org/pdf/1609.08144.pdf) | Inference |  | [FP32](language_translation/tensorflow/mlperf_gnmt/inference/README.md) | [MLPerf GNMT model benchmarking dataset](https://github.com/IntelAI/models/tree/master/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32#datasets) |
-| Language Translation    | [Transformer_LT_mlperf*](https://arxiv.org/pdf/1706.03762.pdf) | Training | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/transformer-lt-mlperf-fp32-training-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/transformer-lt-mlperf-bfloat16-training-tensorflow-container.html) | [FP32 BFloat16](language_translation/tensorflow/transformer_mlperf/training/README.md) | [WMT English-German dataset](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
-| Language Translation    | [Transformer_LT_mlperf*](https://arxiv.org/pdf/1706.03762.pdf) | Inference | | [FP32 BFloat16  Int8](language_translation/tensorflow/transformer_mlperf/inference/README.md) | [WMT English-German data](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
-| Language Translation    | [Transformer_LT_Official](https://arxiv.org/pdf/1706.03762.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/transformer-lt-official-fp32-inference-tensorflow-container.html) | [FP32](language_translation/tensorflow/transformer_lt_official/inference/README.md) | [WMT English-German dataset](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
-| Object Detection        | [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf) | Inference |  | [Int8](object_detection/tensorflow/faster_rcnn/inference/int8/README.md) [FP32](object_detection/tensorflow/faster_rcnn/inference/fp32/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
-| Object Detection        | [R-FCN](https://arxiv.org/pdf/1605.06409.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-fp32-inference-tensorflow-container.html) | [Int8 FP32](object_detection/tensorflow/rfcn/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
-| Object Detection        | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-mobilenet-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-mobilenet-fp32-inference-tensorflow-container.html) | [Int8 FP32 BFloat16](object_detection/tensorflow/ssd-mobilenet/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
-| Object Detection        | [SSD-ResNet34*](https://arxiv.org/pdf/1512.02325.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-resnet34-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-resnet34-fp32-inference-tensorflow-container.html) | [Int8 FP32 BFloat16](object_detection/tensorflow/ssd-resnet34/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
-| Object Detection        | [SSD-ResNet34](https://arxiv.org/pdf/1512.02325.pdf) | Training | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-resnet34-fp32-training-tensorflow-container.html) [BFloat16](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-resnet34-bfloat16-training-tensorflow-container.html) | [FP32](object_detection/tensorflow/ssd-resnet34/training/fp32/README.md) [BFloat16](object_detection/tensorflow/ssd-resnet34/training/bfloat16/README.md) | [COCO 2017 training dataset](https://github.com/IntelAI/models/tree/master/datasets/coco/README_train.md) |
-| Recommendation          | [DIEN](https://arxiv.org/abs/1809.03672) | Inference | | [FP32 BFloat16](/benchmarks/recommendation/tensorflow/dien/inference/README.md) | [DIEN dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/dien/inference#datasets) |
-| Recommendation          | [DIEN](https://arxiv.org/abs/1809.03672) | Training | | [FP32](/benchmarks/recommendation/tensorflow/dien/training/README.md) | [DIEN dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/dien#1-prepare-datasets-1) |
-| Recommendation          | [NCF](https://arxiv.org/pdf/1708.05031.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/ncf-fp32-inference-tensorflow-container.html) | [FP32](/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md) | [MovieLens 1M](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/ncf/inference/fp32#datasets) |
-| Recommendation          | [Wide & Deep](https://arxiv.org/pdf/1606.07792.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-fp32-inference-tensorflow-container.html) | [FP32](/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md) | [Census Income dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/wide_deep/inference/fp32#dataset) |
-| Recommendation          | [Wide & Deep Large Dataset](https://arxiv.org/pdf/1606.07792.pdf) | Inference | Model Containers: [Int8](https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-large-dataset-int8-inference-tensorflow-container.html) [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-large-dataset-fp32-inference-tensorflow-container.html) | [Int8 FP32](/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md) | [Large Kaggle Display Advertising Challenge dataset](https://github.com/IntelAI/models/tree/master/datasets/large_kaggle_advertising_challenge/README.md) |
-| Recommendation          | [Wide & Deep Large Dataset](https://arxiv.org/pdf/1606.07792.pdf) | Training | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-large-dataset-fp32-training-tensorflow-container.html) | [FP32](/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md) | [Large Kaggle Display Advertising Challenge dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32#dataset) |
-| Text-to-Speech          | [WaveNet](https://arxiv.org/pdf/1609.03499.pdf) | Inference | Model Containers: [FP32](https://www.intel.com/content/www/us/en/developer/articles/containers/wavenet-fp32-inference-tensorflow-container.html) | [FP32](/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md) |
-
-## TensorFlow Serving Use Cases
-
-| Use Case               | Model               | Mode      | Model Documentation |
-| -----------------------| ------------------- | --------- |---------------------|
-| Image Recognition      | [Inception V3](https://arxiv.org/pdf/1512.00567.pdf)        | Inference | [FP32](image_recognition/tensorflow_serving/inceptionv3/README.md#fp32-inference-instructions) |
-| Image Recognition      | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | Inference | [FP32](image_recognition/tensorflow_serving/resnet50v1_5/README.md#fp32-inference-instructions) |
-| Language Translation   | [Transformer_LT_Official](https://arxiv.org/pdf/1706.03762.pdf) | Inference | [FP32](language_translation/tensorflow_serving/transformer_lt_official/README.md#fp32-inference-instructions) |
-| Object Detection       | [SSD-MobileNet](https://arxiv.org/pdf/1704.04861.pdf)       | Inference | [FP32](object_detection/tensorflow_serving/ssd-mobilenet/README.md#fp32-inference-instructions) |
-
-## PyTorch Use Cases
-
-| Use Case                | Model              | Mode      | Model Documentation |
-| ----------------------- | --------- | ------------------- | --------------------|
-| Image Recognition       | [GoogLeNet](https://arxiv.org/abs/1409.4842)         | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/googlenet/inference/cpu/README.md) |
-| Image Recognition       | [Inception v3](https://arxiv.org/pdf/1512.00567.pdf) | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/inception_v3/inference/cpu/README.md) |
-| Image Recognition       | [MNASNet 0.5](https://arxiv.org/abs/1807.11626)      | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/mnasnet0_5/inference/cpu/README.md) |
-| Image Recognition       | [MNASNet 1.0](https://arxiv.org/abs/1807.11626)      | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/mnasnet1_0/inference/cpu/README.md) |
-| Image Recognition       | [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)    | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md) |
-| Image Recognition       | [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)    | Training | [FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md) |
-| Image Recognition       | [ResNet 101](https://arxiv.org/pdf/1512.03385.pdf)   | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnet101/inference/cpu/README.md) |
-| Image Recognition       | [ResNet 152](https://arxiv.org/pdf/1512.03385.pdf)   | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnet152/inference/cpu/README.md) |
-| Image Recognition       | [ResNext 32x4d](https://arxiv.org/abs/1611.05431)    | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnext-32x4d/inference/cpu/README.md) |
-| Image Recognition       | [ResNext 32x16d](https://arxiv.org/abs/1611.05431)   | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md) |
-| Image Recognition       | [VGG-11](https://arxiv.org/abs/1409.1556)            | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/vgg11/inference/cpu/README.md) |
-| Image Recognition       | [VGG-11 with batch normalization](https://arxiv.org/abs/1409.1556) | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/vgg11_bn/inference/cpu/README.md) |
-| Image Recognition       | [Wide ResNet-50-2](https://arxiv.org/pdf/1605.07146.pdf)   | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/wide_resnet50_2/inference/cpu/README.md) |
-| Image Recognition       | [Wide ResNet-101-2](https://arxiv.org/pdf/1605.07146.pdf)  | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/wide_resnet101_2/inference/cpu/README.md) |
-| Language Modeling       | [BERT base](https://arxiv.org/pdf/1810.04805.pdf)    | Inference | [FP32 BFloat16](/quickstart/language_modeling/pytorch/bert_base/inference/cpu/README.md) |
-| Language Modeling       | [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md) |
-| Language Modeling       | [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | Training  | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md) |
-| Language Modeling       | [DistilBERT base](https://arxiv.org/abs/1910.01108)  | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md) |
-| Language Modeling       | [RNN-T](https://arxiv.org/abs/2007.15188)            | Inference | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md) |
-| Language Modeling       | [RNN-T](https://arxiv.org/abs/2007.15188)            | Training  | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md) |
-| Language Modeling       | [RoBERTa base](https://arxiv.org/abs/1907.11692)     | Inference | [FP32 BFloat16](/quickstart/language_modeling/pytorch/roberta_base/inference/cpu/README.md) |
-| Language Modeling       | [T5](https://arxiv.org/abs/1910.10683)     | Inference | [FP32 Int8](/quickstart/language_modeling/pytorch/t5/inference/cpu/README.md) |
-| Object Detection        | [Faster R-CNN ResNet50 FPN](https://arxiv.org/abs/1506.01497) | Inference | [FP32 BFloat16](/quickstart/object_detection/pytorch/faster_rcnn_resnet50_fpn/inference/cpu/README.md) |
-| Object Detection        | [Mask R-CNN](https://arxiv.org/abs/1703.06870)                | Inference  | [FP32 BFloat16 BFloat32](/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/README.md) |
-| Object Detection        | [Mask R-CNN](https://arxiv.org/abs/1703.06870)                | Training   | [FP32 BFloat16 BFloat32](/quickstart/object_detection/pytorch/maskrcnn/training/cpu/README.md) |
-| Object Detection        | [Mask R-CNN ResNet50 FPN](https://arxiv.org/abs/1703.06870)   | Inference | [FP32 BFloat16](/quickstart/object_detection/pytorch/maskrcnn_resnet50_fpn/inference/cpu/README.md) |
-| Object Detection        | [RetinaNet ResNet-50 FPN](https://arxiv.org/abs/1708.02002)   | Inference | [FP32 BFloat16](/quickstart/object_detection/pytorch/retinanet_resnet50_fpn/inference/cpu/README.md) |
-| Object Detection        | [SSD-ResNet34](https://arxiv.org/abs/1512.02325)              | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/README.md) |
-| Object Detection        | [SSD-ResNet34](https://arxiv.org/abs/1512.02325)              | Training | [FP32 BFloat16 BFloat32](/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/README.md) |
-| Recommendation          | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md) |
-| Recommendation          | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | Training  | [FP32 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md) |
-| Shot Boundary Detection          | [TransNetV2](https://arxiv.org/pdf/2008.04838.pdf)         | Inference  | [FP32 BFloat16](/quickstart/shot_boundary_detection/pytorch/transnetv2/inference/cpu/README.md) |
-| AI Drug Design (AIDD) | [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2)| Inference  | [FP32](/quickstart/aidd/pytorch/alphafold2/inference/README.md) |
-
-*Means the model belongs to [MLPerf](https://mlperf.org/) models and will be supported long-term.
diff --git a/benchmarks/common/base_benchmark_util.py b/benchmarks/common/base_benchmark_util.py
index 58cd8bbb9..bc254fa20 100644
--- a/benchmarks/common/base_benchmark_util.py
+++ b/benchmarks/common/base_benchmark_util.py
@@ -228,13 +228,6 @@ def _define_args(self):
             dest="experimental_gelu", choices=["True", "False"],
             default=False)
 
-        self._common_arg_parser.add_argument(
-            "--amp",
-            help="use grappler auto-mixed precision as opposed to \
-            keras mixed precision",
-            dest="amp", choices=["True", "False"],
-            default=False)
-
         # Note this can't be a normal boolean flag, because we need to know when the user
         # does not explicitly set the arg value so that we can apply the appropriate
         # default value, depending on the the precision.
@@ -287,6 +280,14 @@ def _define_args(self):
             help="Run the benchmark script using GPU",
             dest="gpu", action="store_true")
 
+        # Check if OneDNN Graph is enabled
+        self._common_arg_parser.add_argument(
+            "--onednn-graph",
+            help="If Intel® Extension for TensorFlow* is installed, oneDNN Graph for INT8 will be enabled"
+                 " by default. Otherwise, default value of this flag will be False.",
+            dest="onednn_graph", choices=["True", "False"],
+            default=None)
+
     def _validate_args(self):
         """validate the args and initializes platform_util"""
         # check if socket id is in socket number range
diff --git a/benchmarks/common/tensorflow/start.sh b/benchmarks/common/tensorflow/start.sh
index 43323f8dc..80d338a6e 100644
--- a/benchmarks/common/tensorflow/start.sh
+++ b/benchmarks/common/tensorflow/start.sh
@@ -55,6 +55,7 @@ echo "    PYTHON_EXE: ${PYTHON_EXE}"
 echo "    PYTHONPATH: ${PYTHONPATH}"
 echo "    DRY_RUN: ${DRY_RUN}"
 echo "    GPU: ${GPU}"
+echo "    ONEDNN_GRAPH: ${ONEDNN_GRAPH}"
 
 #  Enable GPU Flag
 gpu_arg=""
@@ -80,6 +81,13 @@ if [ ${MODE} != "inference" ] && [ ${MODE} != "training" ]; then
   exit 1
 fi
 
+# Enable OneDNN Graph Flag
+onednn_graph_arg=""
+if [ ${ONEDNN_GRAPH} == "True" ]; then
+  onednn_graph_arg="--onednn-graph=True"
+  export ITEX_ONEDNN_GRAPH=1
+fi
+
 # Determines if we are running in a container by checking for .dockerenv
 function _running-in-container()
 {
@@ -408,7 +416,8 @@ ${output_results_arg} \
 ${weight_sharing_arg} \
 ${synthetic_data_arg} \
 ${verbose_arg} \
-${gpu_arg}"
+${gpu_arg} \
+${onednn_graph_arg}"
 
 if [ ${MOUNT_EXTERNAL_MODELS_SOURCE} != "None" ]; then
   CMD="${CMD} --model-source-dir=${MOUNT_EXTERNAL_MODELS_SOURCE}"
@@ -1431,10 +1440,12 @@ function gpt_j() {
         if [ ${BENCHMARK_ONLY} == "True" ]; then
           CMD=" ${CMD} --max_output_tokens=${MAX_OUTPUT_TOKENS}"
           CMD=" ${CMD} --input_tokens=${INPUT_TOKENS}"
-          if [[ -z "${SKIP_ROWS}" ]]; then
-            SKIP_ROWS=0
+          CMD=" ${CMD} --steps=${STEPS}"
+          CMD=" ${CMD} --warmup_steps=${WARMUP_STEPS}"
+          if [[ -z "${DUMMY_DATA}" ]]; then
+            DUMMY_DATA=0
           fi
-          CMD=" ${CMD} --skip_rows=${SKIP_ROWS}"
+          CMD=" ${CMD} --dummy_data=${DUMMY_DATA}"
         fi
         CMD=${CMD} run_model
       else
@@ -1683,6 +1694,7 @@ function vision_transformer() {
 
     if [ ${MODE} == "training" ]; then
 	CMD="${CMD} $(add_arg "--init-checkpoint" ${INIT_CHECKPOINT})"
+	CMD="${CMD} $(add_arg "--epochs" ${EPOCHS})"
     fi
 	    
     if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat16" ] ||
@@ -1775,6 +1787,58 @@ function rgat() {
     fi
 }
 
+function stable_diffusion() {
+    if [ ${MODE} == "inference" ]; then
+      if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat16" ] || [ ${PRECISION} == "fp16" ]; then
+        curr_dir=${pwd}
+        echo "Curr dir: "
+        echo ${curr_dir}
+
+        infer_dir=${MOUNT_INTELAI_MODELS_SOURCE}/${MODE}
+        benchmarks_patch_path=${infer_dir}/patch
+        echo "benchmarks_patch_path:"
+        echo ${benchmarks_patch_path}
+
+        cd /tmp
+        rm -rf keras-cv
+        git clone https://github.com/keras-team/keras-cv.git
+        cd keras-cv
+        git reset --hard 66fa74b6a2a0bb1e563ae8bce66496b118b95200
+        git apply ${benchmarks_patch_path}
+        pip install .
+        cd ${curr_dir}
+
+        if [[ ${NOINSTALL} != "True" ]]; then
+          python3 -m pip install -r "${MOUNT_BENCHMARK}/${USE_CASE}/${FRAMEWORK}/${MODEL_NAME}/${MODE}/requirements.txt"
+        fi
+
+        python -c $'from tensorflow import keras\n_ = keras.utils.get_file(
+            "bpe_simple_vocab_16e6.txt.gz",
+            "https://github.com/openai/CLIP/blob/main/clip/bpe_simple_vocab_16e6.txt.gz?raw=true",
+            file_hash="924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a",
+        )\n_ = keras.utils.get_file(
+          origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_encoder.h5",
+          file_hash="4789e63e07c0e54d6a34a29b45ce81ece27060c499a709d556c7755b42bb0dc4",
+        )\n_ = keras.utils.get_file(
+          origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_diffusion_model.h5",
+          file_hash="8799ff9763de13d7f30a683d653018e114ed24a6a819667da4f5ee10f9e805fe",
+        )\n_ = keras.utils.get_file(
+          origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_decoder.h5",
+          file_hash="ad350a65cc8bc4a80c8103367e039a3329b4231c2469a1093869a345f55b1962",
+        )'
+
+        export PYTHONPATH=${PYTHONPATH}:${MOUNT_EXTERNAL_MODELS_SOURCE}
+
+        CMD="${CMD} $(add_arg "--steps" ${STEPS})"
+        CMD="${CMD} $(add_arg "--output-dir" ${OUTPUT_DIR})"
+        CMD=${CMD} run_model
+      else
+        echo "PRECISION=${PRECISION} not supported for ${MODEL_NAME} in this repo."
+        exit 1
+      fi
+    fi
+}
+
 # Wide & Deep model
 function wide_deep() {
     if [ ${PRECISION} == "fp32" ]; then
@@ -1857,7 +1921,7 @@ function wide_deep_large_ds() {
 
 function graphsage() {
     if [ ${MODE} == "inference" ]; then
-      if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat16" ] || [ ${PRECISION} == "fp16" ]; then
+      if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat16" ] || [ ${PRECISION} == "fp16" ] || [ ${PRECISION} == "int8" ]; then
         export PYTHONPATH=${PYTHONPATH}:${MOUNT_EXTERNAL_MODELS_SOURCE}
 
         if [ ${NUM_INTER_THREADS} != "None" ]; then
@@ -1955,6 +2019,8 @@ elif [ ${MODEL_NAME} == "gpt_j" ]; then
   gpt_j
 elif [ ${MODEL_NAME} == "rgat" ]; then
   rgat
+elif [ ${MODEL_NAME} == "stable_diffusion" ]; then
+  stable_diffusion
 else
   echo "Unsupported model: ${MODEL_NAME}"
   exit 1
diff --git a/benchmarks/image_recognition/tensorflow/densenet169/inference/README.md b/benchmarks/image_recognition/tensorflow/densenet169/inference/README.md
index 84d5cb370..1c750ec3f 100644
--- a/benchmarks/image_recognition/tensorflow/densenet169/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/densenet169/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to this directory when running DenseNet 169.
 | [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=100). |
 | [`accuracy.sh`](/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -82,7 +82,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset>
@@ -99,7 +99,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using a `cmd.exe` run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/densenet169/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/densenet169/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 9271db5ec..000000000
--- a/benchmarks/image_recognition/tensorflow/densenet169/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,107 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-Run throughput benchmarking with batch size 100 using the following command:
-```
-python launch_benchmark.py \
-    --model-name densenet169 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --in-graph ${PRETRAINED_MODEL} \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --output-dir ${OUTPUT_DIR} \
-    -- input_height=224 input_width=224 warmup_steps=20 steps=100 \
-    input_layer="input" output_layer="densenet169/predictions/Reshape_1"
-```
-
-Run latency benchmarking with batch size 1 using the following command:
-```
-python launch_benchmark.py \
-    --model-name densenet169 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --socket-id 0 \
-    --in-graph ${PRETRAINED_MODEL} \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --output-dir ${OUTPUT_DIR} \
-    -- input_height=224 input_width=224 warmup_steps=20 steps=100 \
-    input_layer="input" output_layer="densenet169/predictions/Reshape_1"
-```
-
-Run an accuracy test with the following command:
-```
-python launch_benchmark.py \
-    --model-name densenet169 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --accuracy-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --in-graph ${PRETRAINED_MODEL} \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    -- input_height=224 input_width=224 \
-    input_layer="input" output_layer="densenet169/predictions/Reshape_1"
-```
-
-Output files and logs are saved to the `${OUTPUT_DIR}` directory. Below are
-examples of what the tail of your log file should look like for the different configs.
-
-Example log tail when running for batch inference:
-```
-steps = 80, 159.83471377 images/sec
-       Latency: 625.646317005 ms
-steps = 90, 159.852789241 images/sec
-       Latency: 625.57557159 ms
-steps = 100, 159.853966416 images/sec
-       Latency: 625.570964813 ms
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_densenet169_inference_fp32_20190412_023940.log
-```
-
-Example log tail when running for online inference:
-```
-steps = 80, 34.9948442873 images/sec
-       Latency: 28.5756379366 ms
-steps = 90, 34.9644341907 images/sec
-       Latency: 28.6004914178 ms
-steps = 100, 34.9655988121 images/sec
-       Latency: 28.5995388031 ms
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_densenet169_inference_fp32_20190412_024505.log
-```
-
-Example log tail when running for accuracy:
-```
-Iteration time: 581.6446 ms
-0.757505030181
-Iteration time: 581.5755 ms
-0.757489959839
-Iteration time: 581.5709 ms
-0.75749498998
-Iteration time: 581.1705 ms
-0.75748
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_densenet169_inference_fp32_20190412_021545.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/README.md b/benchmarks/image_recognition/tensorflow/inceptionv3/inference/README.md
index dd9825943..ddaf9c356 100644
--- a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/inceptionv3/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to this directory when running Inception V3.
 | [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=128). |
 | [`accuracy.sh`](/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -97,7 +97,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 ### Run on Linux
 
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -113,7 +113,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using a `cmd.exe` run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/inceptionv3/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index d2ce72f48..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,115 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size=100 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 128.
-
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-Example log tail when running for accuracy:
-```
-Iteration time: ... ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7673, 0.9341)
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7674, 0.9341)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7675, 0.9342)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_fp32_20190104_023816.log
-```
-
-Example log tail when running for batch inference:
-```
-Iteration 1: ... sec
-Iteration 2: ... sec
-Iteration 3: ... sec
-...
-Iteration 38: ... sec
-Iteration 39: ... sec
-Iteration 40: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_fp32_20190104_024842.log
-```
-
-Example log tail when running for online inference:
-
-```
-Iteration 1: ... sec
-Iteration 2: ... sec
-Iteration 3: ... sec
-...
-Iteration 38: ... sec
-Iteration 39: ... sec
-Iteration 40: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_fp32_20190104_025220.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/inceptionv3/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index be18fb87f..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv3/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,105 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-  
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 128.
-
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name inceptionv3 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --docker-image <docker image> \
-  --socket-id 0
-```
-
-Example log tail when running for accuracy:
-
-```
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7666, 0.9333)
-Executing command: python /workspace/intelai_models/int8/accuracy.py --input_height=299 --input_width=299 --num_intra_threads=56 --num_inter_threads=2 --batch_size=100 --input_graph=/in_graph/inceptionv3_int8_pretrained_model.pb --data_location=/dataset
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_int8_20190104_013246.log
-```
-
-Example log tail when running for online inference:
-
-```
-...
-steps = 470, ... images/sec
-steps = 480, ... images/sec
-steps = 490, ... images/sec
-steps = 500, ... images/sec
-Average throughput for batch size 1: ... images/sec
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_int8_20190415_220455.log
-```
-
-Example log tail when running for batch inference:
-
-```
-...
-steps = 470, ... images/sec
-steps = 480, ... images/sec
-steps = 490, ... images/sec
-steps = 500, ... images/sec
-Average throughput for batch size 128: ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_inceptionv3_inference_int8_20190416_162155.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md
index a31f818dc..98033dbb5 100644
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to this directory when running Inception V4.
 | [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=240). |
 | [`accuracy.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -88,7 +88,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -104,7 +104,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe` run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index f615eb7bb..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,120 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset (only required for accuracy)>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-Inception V4 can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --accuracy-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
- * For batch inference, use the command below that has a batch size of 240
-   and the `--benchmark-only` flag. Synthetic data will be used, since no
-   dataset is being provided.
-   ```
-   python launch_benchmark.py \
-     --model-name inceptionv4 \
-     --precision fp32 \
-     --mode inference \
-     --framework tensorflow \
-     --benchmark-only \
-     --batch-size 240 \
-     --socket-id 0 \
-     --docker-image <docker image> \
-     --in-graph ${PRETRAINED_MODEL} \
-     --output-dir ${OUTPUT_DIR}
-   ```
-
-* For online inference use the command below that has a batch size of 1 and
-  the `--benchmark-only` flag. Again, synthetic data is being used in this
-  example.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-The log file is saved to the `OUTPUT_DIR` directory. Below are examples of
-what the tail of your log file should look like for the different configs.
-
-Example log tail when running for accuracy:
-```
-...
-Iteration time: 1337.8728 ms
-Processed 49600 images. (Top1 accuracy, Top5 accuracy) = (0.8015, 0.9517)
-Iteration time: 1331.8253 ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.8017, 0.9518)
-Iteration time: 1339.1553 ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.8017, 0.9518)
-Iteration time: 1334.5991 ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.8018, 0.9519)
-Iteration time: 1336.1905 ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.8018, 0.9519)
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190308_182729.log
-```
-
-Example log tail when running for batch inference:
-```
-[Running warmup steps...]
-steps = 10, 91.4372832625 images/sec
-[Running benchmark steps...]
-steps = 10, 91.0217283977 images/sec
-steps = 20, 90.8331507586 images/sec
-steps = 30, 91.1284943026 images/sec
-steps = 40, 91.1885998597 images/sec
-steps = 50, 91.1905741783 images/sec
-Ran inference with batch size 240
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190308_184431.log
-```
-
-Example log tail when running for online inference:
-```
-[Running warmup steps...]
-steps = 10, 15.6993019295 images/sec
-[Running benchmark steps...]
-steps = 10, 16.3553780883 images/sec
-steps = 20, 15.771143231 images/sec
-steps = 30, 15.7133587586 images/sec
-steps = 40, 16.0477494988 images/sec
-steps = 50, 15.483992912 images/sec
-Latency: 63.534 ms
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190307_221954.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index aeef6f8d5..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,117 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-Inception V4 can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* Use the command below to test accuracy. This command uses the `DATASET_DIR`,
-  a batch size of 100 and the `--accuracy-only` flag:
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --accuracy-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-* For batch inference, use the command below that uses a batch size of 240 and
-  the `--benchmark-only` flag. Since no dataset is being provided, synthetic data
-  will be used.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 240 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-* For online inference, use the command below that has a batch size of 1 and the
-  `--benchmark-only` flag. Again, synthetic data is being used in this example.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-The log file is saved to the `OUTPUT_DIR` directory. Below are examples of
-what the tail of your log file should look like for the different configs.
-
-Example log tail when running for accuracy:
-```
-...
-Iteration time: 685.1976 ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7985, 0.9504)
-Iteration time: 686.3845 ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7983, 0.9504)
-Iteration time: 686.7021 ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7984, 0.9504)
-Iteration time: 685.8914 ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7984, 0.9504)
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190306_221608.log
-```
-
-Example log tail when running for batch inference:
-```
-[Running warmup steps...]
-steps = 10, 184.497605972 images/sec
-[Running benchmark steps...]
-steps = 10, 184.664702184 images/sec
-steps = 20, 184.938455688 images/sec
-steps = 30, 184.454197634 images/sec
-steps = 40, 184.491891402 images/sec
-steps = 50, 184.390001575 images/sec
-Ran inference with batch size 240
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190415_233517.log
-```
-
-Example log tail when running for online inference:
-```
-[Running warmup steps...]
-steps = 10, 32.6095380262 images/sec
-[Running benchmark steps...]
-steps = 10, 32.9024373024 images/sec
-steps = 20, 32.5328989723 images/sec
-steps = 30, 32.5988932413 images/sec
-steps = 40, 31.3991914957 images/sec
-steps = 50, 32.7053998207 images/sec
-Latency: 30.598 ms
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190415_232441.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/README.md b/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/README.md
index af8e44e28..a5d7f50bd 100644
--- a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to the TF records directory when running MobileNe
 | `inference_realtime_multi_instance.sh` | A multi-instance run that uses 4 cores per instance with `batch_size=1` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set|
 | `inference_throughput_multi_instance.sh` | A multi-instance run that uses 4 cores per instance with `batch_size=448` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set |
 | `accuracy.sh` | Measures the model accuracy (batch_size=100) for the specified precision (fp32, int8, bfloat16, bfloat32). |
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -95,7 +95,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -111,7 +111,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe` run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/bfloat16/README.md b/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/bfloat16/README.md
deleted file mode 100644
index 00a4633bd..000000000
--- a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/bfloat16/README.md
+++ /dev/null
@@ -1,97 +0,0 @@
-<!--- 0. Title -->
-# MobileNet V1 BFloat16 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running MobileNet V1 BFloat16 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-Set the `DATASET_DIR` to point to the TF records directory when running <model name>.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, int8 or bfloat16) with 1000 steps, 500 warmup steps and `batch-size=1`. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value. |
-| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, int8 or bfloat16) with 1000 steps, 500 warmup steps and `batch-size=448`. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value. |
-| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, int8 or bfloat16) with `batch-size=100`. |
-
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the tensorflow conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After finishing the setup above, download the pretrained model and set the
-`PRETRAINED_MODEL` environment var to the path to the frozen graph.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
-export PRETRAINED_MODEL=$(pwd)/mobilenet_v1_1.0_224_frozen.pb
-```
-Set environment variables for the path to your `DATASET_DIR` for ImageNet
-and an `OUTPUT_DIR` where log files will be written. Navigate to your
-model zoo directory and then run a [quickstart script](#quick-start-scripts).
-
-### Run on Linux:
-```
-# cd to your model zoo directory
-cd models
-
-# Set environment variables
-export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-export DATASET_DIR=<path to the ImageNet TF records>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRECISION=bfloat16
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script (for example, inference realtime.sh)
-./quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
- To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
-
-
diff --git a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 0896d027a..000000000
--- a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,157 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --docker-image <docker image> \
-  --batch-size 100
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 100.
-
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image> \
-  -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
-  input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --socket-id 0 \
-  --docker-image <docker image> \
-  -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
-  input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
-```
-
-Example log file snippet when testing accuracy:
-```
-Iteration time: ... ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7104, 0.8999)
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7103, 0.8999)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7102, 0.8999)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_fp32_20190110_211648.log
-```
-
-Example log file snippet when testing batch inference:
-```
-2019-01-04 20:06:01.151312: step 80, ... images/sec
-2019-01-04 20:06:06.719081: step 90, ... images/sec
-2019-01-04 20:06:12.346302: step 100, ... images/sec
-eval/Accuracy[0.0009]
-eval/Recall_5[0.0049]
-INFO:tensorflow:Finished evaluation at 2019-01-04-20:06:12
-self._total_images_per_sec = ...
-self._displayed_steps = 10
-Total images/sec = ...
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_fp32_20190104_200512.log
-```
-
-Example log file snippet when testing online inference:
-```
-2019-01-04 20:02:23.855441: step 80, ... images/sec
-2019-01-04 20:02:23.974862: step 90, ... images/sec
-2019-01-04 20:02:24.097476: step 100, ... images/sec
-eval/Accuracy[0]
-eval/Recall_5[0]
-INFO:tensorflow:Finished evaluation at 2019-01-04-20:02:24
-self._total_images_per_sec = ...
-self._displayed_steps = 10
-Total images/sec = ...
-Latency ms/step = ...
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_fp32_20190104_200218.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with synthetic data (to use real data, you can add `--data-location ${DATASET_DIR}`).
-Your output may vary from what's seen below, depending on the number of
-cores on your system.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 56.
-  ```  
-  python launch_benchmark.py \
-    --model-name mobilenet_v1 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 56 \
-    --numa-cores-per-instance socket \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    -- input_height=224 input_width=224 warmup_steps=500 steps=1000 \
-    input_layer='input' output_layer='MobilenetV1/Predictions/Reshape_1'
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name mobilenet_v1 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    -- input_height=224 input_width=224 warmup_steps=500 steps=1000 \
-    input_layer='input' output_layer='MobilenetV1/Predictions/Reshape_1'
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 889231cae..000000000
--- a/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,159 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --accuracy-only \
-  --batch-size 100  \
-  --socket-id 0 \
-  --output-dir ${OUTPUT_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --docker-image <docker image> \
-  -- input_height=224 input_width=224 \
-  input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 240, and the `--benchmark-only` flag.
-
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --benchmark-only \
-  --batch-size 240  \
-  --socket-id 0 \
-  --output-dir ${OUTPUT_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --docker-image <docker image> \
-  -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
-  input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 1, and the `--benchmark-only` flag.
-  
-```
-python launch_benchmark.py \
-  --model-name mobilenet_v1 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --benchmark-only \
-  --batch-size 1  \
-  --socket-id 0 \
-  --output-dir ${OUTPUT_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --docker-image <docker image> \
-  -- input_height=224 input_width=224 warmup_steps=10 steps=50 \
-  input_layer="input" output_layer="MobilenetV1/Predictions/Reshape_1"
-```
-
-Example log tail when running for accuracy:
-```
-Iteration time: ... ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7014, 0.8935)
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7014, 0.8934)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7014, 0.8934)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_int8_20190523_164955.log
-```
-
-Example log tail when running for batch inference:
-```
-[Running warmup steps...]
-steps = 10, ... images/sec
-[Running benchmark steps...]
-steps = 10, ... images/sec
-steps = 20, ... images/sec
-steps = 30, ... images/sec
-steps = 40, ... images/sec
-steps = 50, ... images/sec
-Ran inference with batch size 240
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_int8_20190523_164626.log
-```
-
-Example log tail when running for online inference:
-```
-[Running warmup steps...]
-steps = 10, ... images/sec
-[Running benchmark steps...]
-steps = 10, ... images/sec
-steps = 20, ... images/sec
-steps = 30, ... images/sec
-steps = 40, ... images/sec
-steps = 50, ... images/sec
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_mobilenet_v1_inference_int8_20190523_164348.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with synthetic data (to use real data, you can add `--data-location ${DATASET_DIR}`).
-Your output may vary from what's seen below, depending on the number of
-cores on your system.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 56.
-  ```  
-  python launch_benchmark.py \
-    --model-name mobilenet_v1 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 56 \
-    --numa-cores-per-instance socket \
-    --output-dir ${OUTPUT_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    -- input_height=224 input_width=224 warmup_steps=500 steps=1000 \
-    input_layer='input' output_layer='MobilenetV1/Predictions/Reshape_1'
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name mobilenet_v1 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --output-dir ${OUTPUT_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    -- input_height=224 input_width=224 warmup_steps=500 steps=1000 \
-    input_layer='input' output_layer='MobilenetV1/Predictions/Reshape_1'
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/mobilenet_v2/inference/README.md b/benchmarks/image_recognition/tensorflow/mobilenet_v2/inference/README.md
index b70a4ad18..9edc77ae8 100644
--- a/benchmarks/image_recognition/tensorflow/mobilenet_v2/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/mobilenet_v2/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to the TF records directory when running MobileNe
 | `inference_realtime_multi_instance.sh` | A multi-instance run that uses 4 cores per instance with `batch_size=1` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set|
 | `inference_throughput_multi_instance.sh` | A multi-instance run that uses 4 cores per instance with `batch_size=448` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set |
 | `accuracy.sh` | Measures the model accuracy (batch_size=100) for the specified precision (fp32, int8, bfloat16, bfloat32). |
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -90,7 +90,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -106,7 +106,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe` run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/resnet101/inference/README.md b/benchmarks/image_recognition/tensorflow/resnet101/inference/README.md
index a41e19022..df3d59c0b 100644
--- a/benchmarks/image_recognition/tensorflow/resnet101/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/resnet101/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to this directory when running ResNet101.
 | [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=128). |
 | [`accuracy.sh`](/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -95,7 +95,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -111,7 +111,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/resnet101/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet101/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 0437c7a1c..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet101/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,90 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 128.
-
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-Example log file snippet when testing accuracy:
-```
-Processed 49600 images. (Top1 accuracy, Top5 accuracy) = (0.7639, 0.9289)
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7639, 0.9290)
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7639, 0.9289)
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7641, 0.9289)
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7640, 0.9289)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_resnet101_inference_fp32_20190104_201506.log
-```
-
-Example log file snippet when testing batch or online inference:
-```
-steps = 70, ... images/sec
-steps = 80, ... images/sec
-steps = 90, ... images/sec
-steps = 100, ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_resnet101_inference_fp32_20190104_204615.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/resnet101/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet101/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 5b907081a..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet101/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,106 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* When running for performance, it is optional to specify the
-  number of `warmup_steps` and `steps` as extra args, as shown in the
-  commands below. If these values are not specified, the script will
-  default to use `warmup_steps=40` and `steps=100`. For batch inference, 
-  use the command below that uses the `DATASET_DIR`, a batch size of 128,
-  `warmup_steps=50`, and `steps=500`.
-
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --socket-id 0 \
-  --docker-image <docker image> \
-  -- warmup_steps=50 steps=500
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 1, `warmup_steps=50`, and `steps=500`.
-  
-```
-python launch_benchmark.py \
-  --model-name resnet101 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --docker-image <docker image> \
-  --socket-id 0 \
-  -- warmup_steps=50 steps=500
-```
-
-Example log file snippet when testing accuracy:
-```
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7690, 0.9304)
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7691, 0.9305)
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7691, 0.9305)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_resnet101_inference_int8_20190104_205838.log
-```
-
-Example log file snippet when testing batch inference:
-```
-...
-steps = 470, ... images/sec
-steps = 480, ... images/sec
-steps = 490, ... images/sec
-steps = 500, ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_resnet101_inference_int8_20190223_192438.log
-```
-
-Example log file snippet when testing online inference:
-```
-...
-steps = 470, ... images/sec
-steps = 480, ... images/sec
-steps = 490, ... images/sec
-steps = 500, ... images/sec
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_resnet101_inference_int8_20190223_191406.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50/inference/README.md b/benchmarks/image_recognition/tensorflow/resnet50/inference/README.md
index 7181ef6ed..dce76d42a 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/resnet50/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to this directory when running ResNet50.
 
 
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -48,22 +48,22 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -95,7 +95,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -111,7 +111,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/resnet50/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index a96fd12db..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,112 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name resnet50 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 128.
-
-```
-python launch_benchmark.py \
-  --model-name resnet50 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name resnet50 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 1 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-Example log file snippet when testing accuracy:
-```
-...
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7430, 0.9188)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_213452.log
-```
-
-Example log file snippet when testing batch inference:
-```
-Inference with dummy data.
-Iteration 1: ... sec
-Iteration 2: ... sec
-Iteration 3: ... sec
-...
-Iteration 38: ... sec
-Iteration 39: ... sec
-Iteration 40: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_215655.log
-```
-
-Example log file snippet when testing online inference:
-```
-Inference with dummy data.
-Iteration 1: ... sec
-Iteration 2: ... sec
-...
-Iteration 38: ... sec
-Iteration 39: ... sec
-Iteration 40: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_215326.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 30eeee145..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,84 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name resnet50 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 128, 500 steps, and 50 warmup_steps.
-
-```
-python launch_benchmark.py \
-  --model-name resnet50 \
-  --precision int8 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size 128 \
-  --benchmark-only \
-  --socket-id 0 \
-  --docker-image <docker image>
-  -- warmup_steps=50 steps=500
-```
-
-Example log file snippet when testing accuracy:
-```
-Iteration time: ... ms
-Processed 49600 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
-Iteration time: ... ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
-Iteration time: ... ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7360, 0.9154)
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7360, 0.9154)
-Ran inference with batch size 100
-Log location outside container: {--output-dir value}/benchmark_resnet50_inference_int8_20190104_212224.log
-```
-
-Example log file snippet when testing batch inference:
-```
-...
-Iteration 497: ... sec
-Iteration 498: ... sec
-Iteration 499: ... sec
-Iteration 500: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log location outside container: {--output-dir value}/benchmark_resnet50_inference_int8_20190416_172735.log
-```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/README.md b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/README.md
index 62590f2a5..c130f79fd 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/README.md
+++ b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/README.md
@@ -27,21 +27,21 @@ Set the `DATASET_DIR` to point to the TF records directory when running ResNet50
 | `inference_throughput_multi_instance.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, int8, bfloat16, fp16, bfloat32) with 1500 steps and 50 warmup steps. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value. |
 | `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, int8, bfloat16, fp16, bfloat32). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -51,12 +51,13 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>openssh-server (only required for multi-instance)
         <li>libopenmpi-dev (only required for multi-instance)
         <li>horovod==0.27.0 (only required for multi-instance)
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
         <li>Activate the tensorflow conda environment
         <pre>conda activate tensorflow</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
@@ -69,15 +70,16 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>openssh-server (only required for multi-instance)
         <li>libopenmpi-dev (only required for multi-instance)
         <li>horovod==0.27.0 (only required for multi-instance)
-        <li>A clone of the Model Zoo repo<br />
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -97,6 +99,10 @@ export PRETRAINED_MODEL=$(pwd)/resnet50_v1.pb
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50v1_5_int8_pretrained_model.pb
 export PRETRAINED_MODEL=$(pwd)/resnet50v1_5_int8_pretrained_model.pb
 
+# Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point. Int8 Pretrained model for OneDNN Graph:
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/rn50_itex_int8.pb
+export PRETRAINED_MODEL=$(pwd)/rn50_itex_int8.pb
+
 #BFloat16 Pretrained model:
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_v1_5_bfloat16.pb
 export PRETRAINED_MODEL=$(pwd)/resnet50_v1_5_bfloat16.pb
@@ -106,7 +112,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
@@ -122,7 +128,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 5798e8731..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,147 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision bfloat16 \
-  --mode inference \
-  --batch-size=100 \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 128, and the `--benchmark-only` flag:
-
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision bfloat16 \
-  --mode inference \
-  --batch-size=128 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image>
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 1, and the `--benchmark-only` flag:
-  
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision bfloat16 \
-  --mode inference \
-  --batch-size=1 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image>
-```
-
-Example log file snippet when testing accuracy:
-```
-...
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7672, 0.9314)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7672, 0.9314)
-Ran inference with batch size 100
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_bfloat16_{timestamp}.log
-```
-
-Example log file snippet when testing batch inference:
-```
-...
-Iteration 48: ... sec
-Iteration 49: ... sec
-Iteration 50: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_bfloat16_{timestamp}.log
-```
-
-Example log file snippet when testing online inference:
-```
-...
-Iteration 48: ... sec
-Iteration 49: ... sec
-Iteration 50: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_bfloat16_{timestamp}.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with real data (specified by `--data-location ${DATASET_DIR}`).
-To use synthetic data, you can omit that argument.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 128.
-  ```  
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision bfloat16 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 128 \
-    --numa-cores-per-instance socket \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision bfloat16 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/model_init.py b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/model_init.py
index 5f78e4bef..cf0fa7421 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/model_init.py
+++ b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/bfloat16/model_init.py
@@ -117,6 +117,11 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.data_location and os.listdir(self.args.data_location):
             self.benchmark_command += " --data-location=" + \
                                       self.args.data_location
+
+        # enable onednn graph
+        if self.args.onednn_graph:
+            self.benchmark_command += " --onednn-graph"
+
         if self.args.accuracy_only:
             self.benchmark_command += " --accuracy-only"
 
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 91017ff4f..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,147 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision fp16 \
-  --mode inference \
-  --batch-size=100 \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 128, and the `--benchmark-only` flag:
-
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision fp16 \
-  --mode inference \
-  --batch-size=128 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image>
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 1, and the `--benchmark-only` flag:
-  
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision fp16 \
-  --mode inference \
-  --batch-size=1 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image>
-```
-
-Example log file snippet when testing accuracy:
-```
-...
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7672, 0.9314)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7672, 0.9314)
-Ran inference with batch size 100
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp16_{timestamp}.log
-```
-
-Example log file snippet when testing batch inference:
-```
-...
-Iteration 48: ... sec
-Iteration 49: ... sec
-Iteration 50: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp16_{timestamp}.log
-```
-
-Example log file snippet when testing online inference:
-```
-...
-Iteration 48: ... sec
-Iteration 49: ... sec
-Iteration 50: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp16_{timestamp}.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with real data (specified by `--data-location ${DATASET_DIR}`).
-To use synthetic data, you can omit that argument.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 128.
-  ```  
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision fp16 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 128 \
-    --numa-cores-per-instance socket \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision fp16 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/model_init.py b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/model_init.py
index 7c92f427e..2ed2ba0fc 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/model_init.py
+++ b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp16/model_init.py
@@ -117,6 +117,11 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.data_location and os.listdir(self.args.data_location):
             self.benchmark_command += " --data-location=" + \
                                       self.args.data_location
+
+        # enable onednn graph
+        if self.args.onednn_graph:
+            self.benchmark_command += " --onednn-graph"
+
         if self.args.accuracy_only:
             self.benchmark_command += " --accuracy-only"
 
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 360061450..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,154 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --model-name resnet50v1_5 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --batch-size 100 \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 128.
-
-```
-python launch_benchmark.py \
-  --model-name resnet50v1_5 \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size=128 \
-  --socket-id 0 \
-  --docker-image <docker image> \
-  -- warmup_steps=50 steps=1500
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR` and a batch 
-  size of 1.
-  
-```
-python launch_benchmark.py \
-  --model-name=resnet50v1_5 \
-  --precision=fp32 \
-  --mode=inference \
-  --framework tensorflow \
-  --in-graph ${PRETRAINED_MODEL} \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --batch-size=1 \
-  --socket-id 0 \
-  --docker-image <docker image> \
-  -- warmup_steps=50 steps=1500
-```
-
-Example log file snippet when testing accuracy:
-```
-...
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7648, 0.9308)
-Ran inference with batch size 100
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp32_{timestamp}.log
-```
-
-Example log file snippet when testing batch inference:
-```
-Iteration 1: ... sec
-Iteration 2: ... sec
-Iteration 3: ... sec
-...
-Iteration 388: ... sec
-Iteration 389: ... sec
-Iteration 390: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp32_{timestamp}.log
-```
-
-Example log file snippet when testing online inference:
-```
-Iteration 1: ... sec
-Iteration 2: ... sec
-Iteration 3: ... sec
-...
-Iteration 1498: ... sec
-Iteration 1499: ... sec
-Iteration 1500: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_fp32_{timestamp}.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with real data (specified by `--data-location ${DATASET_DIR}`).
-To use synthetic data, you can omit that argument.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 128.
-  ```  
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 128 \
-    --numa-cores-per-instance socket \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/model_init.py b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/model_init.py
index 756d4aa63..4bebcfb2a 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/model_init.py
+++ b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/fp32/model_init.py
@@ -109,6 +109,11 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.data_location and os.listdir(self.args.data_location):
             self.benchmark_command += " --data-location=" + \
                                       self.args.data_location
+
+        # enable onednn graph
+        if self.args.onednn_graph:
+            self.benchmark_command += " --onednn-graph"
+
         if self.args.accuracy_only:
             self.benchmark_command += " --accuracy-only"
 
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 525071288..000000000
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,152 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-<model name> <precision> <mode> can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-
-```
-python launch_benchmark.py \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision int8 \
-  --mode inference \
-  --batch-size=100 \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only \
-  --socket-id 0 \
-  --docker-image <docker image>
-```
-
-* For batch inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 128, and the `--benchmark-only` flag:
-
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision int8 \
-  --mode inference \
-  --batch-size=128 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image> \
-  -- warmup_steps=50 steps=1500
-```
-
-* For online inference, use the command below that uses the `DATASET_DIR`, a batch 
-  size of 1, and the `--benchmark-only` flag:
-  
-```
-python launch_benchmark.py \
-  --in-graph ${PRETRAINED_MODEL} \
-  --model-name resnet50v1_5 \
-  --framework tensorflow \
-  --precision int8 \
-  --mode inference \
-  --batch-size=1 \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --benchmark-only \
-  --docker-image <docker image> \
-  -- warmup_steps=50 steps=1500
-```
-
-Example log file snippet when testing accuracy:
-```
-Iteration time: ... ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7622, 0.9296)
-Iteration time: ... ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7621, 0.9295)
-Iteration time: ... ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7622, 0.9296)
-Iteration time: ... ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7623, 0.9296)
-Ran inference with batch size 100
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_int8_{timestamp}.log
-```
-
-Example log file snippet when testing batch inference:
-```
-...
-Iteration 48: ... sec
-Iteration 49: ... sec
-Iteration 50: ... sec
-Average time: ... sec
-Batch size = 128
-Throughput: ... images/sec
-Ran inference with batch size 128
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_int8_{timestamp}.log
-```
-
-Example log file snippet when testing online inference:
-```
-...
-Iteration 1499: ... sec
-Iteration 1500: ... sec
-Average time: ... sec
-Batch size = 1
-Latency: ... ms
-Throughput: ... images/sec
-Ran inference with batch size 1
-Log file location: {--output-dir value}/benchmark_resnet50v1_5_inference_int8_{timestamp}.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with real data (specified by `--data-location ${DATASET_DIR}`).
-To use synthetic data, you can omit that argument.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 128.
-  ```  
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 128 \
-    --numa-cores-per-instance socket \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
-  
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --model-name resnet50v1_5 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image <docker image> \
-    -- warmup_steps=50 steps=1500
-  ```
diff --git a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/model_init.py b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/model_init.py
index a7a89e0b9..375d98cc1 100644
--- a/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/model_init.py
+++ b/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/int8/model_init.py
@@ -112,6 +112,11 @@ def run_benchmark_or_accuracy(self):
         # if the data location directory is not empty, then include the arg
         if self.args.data_location and os.listdir(self.args.data_location):
             cmd += " --data-location=" + self.args.data_location
+
+        # enable onednn graph
+        if self.args.onednn_graph:
+            cmd += " --onednn-graph"
+
         if self.args.accuracy_only:
             cmd += " --accuracy-only"
         self.run_command(cmd)
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 74103681c..000000000
--- a/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,39 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model>
-```
-
-<model name> <precision> <mode> can be run with a batch size of 1:
-```
-python launch_benchmark.py \
-  --precision fp32 \
-  --model-name 3d_unet \
-  --mode inference \
-  --framework tensorflow \
-  --docker-image <docker image> \
-  --in-graph $PRETRAINED_MODEL \
-  --data-location ${DATASET_DIR} \
-  --batch-size 1 \
-  --socket-id 0 \
-  --output-dir ${OUTPUT_DIR} 
-```
-
-Below is an example tail of the log file:
-
-```
-Loading pre-trained model
-Time spent per BATCH: ... ms
-Total samples/sec: ... samples/s
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_3d_unet_inference_fp32_20190116_234659.log
-```
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/README.md b/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/README.md
index 25757faa9..c8ee86320 100644
--- a/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/README.md
+++ b/benchmarks/image_segmentation/tensorflow/3d_unet/inference/fp32/README.md
@@ -1,5 +1,5 @@
 <!--- 0. Title -->
-# 3D U-Net FP32 inference
+# 3D U-Net FP32 Inference
 
 <!-- 10. Description -->
 ## Description
@@ -73,23 +73,23 @@ export DATASET_DIR=/home/<user>/3DUnetCNN/brats
 |-------------|-------------|
 | [fp32_inference.sh](fp32_inference.sh) | Runs inference with a batch size of 1 using the BraTS dataset and a pretrained model |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.6 or 3.7
         <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
@@ -101,7 +101,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>nibabel==2.3.3
         <li>SimpleITK===1.2.0
         <li>h5py==2.10.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -123,7 +123,7 @@ the `PRETRAINED_MODEL` path set from the previous command.
 Once the environment variables are all set, you can run the
 [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset>
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md
index 78bfeff96..325f76d30 100644
--- a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md
+++ b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/README.md
@@ -24,32 +24,32 @@ Set the `DATASET_DIR` to point to the directory that contains the dataset files
 | `inference_throughput_multi_instance.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (int8, fp32 or bfloat16) with 100 steps and 50 warmup steps. Dummy data is used for performance evaluation. Waits for all instances to complete, then prints a summarized throughput value. |
 | `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (int8, fp32 or bfloat16). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md) on Linux or Windows systems.
+using [AI Tools](/docs/general/tensorflow/AITools.md) on Linux or Windows systems.
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>Activate the tensorflow conda environment
         <pre>conda activate tensorflow</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -59,6 +59,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md) on Linux or Windows systems.
 Download the pre-trained model based on precision:
 * Download the [FP32 and BFloat16 pre-trained model](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/3dunet_dynamic_ndhwc.pb). 3D-Unet BFloat16 inference depends on Auto-Mixed_precision to convert graph from FP32 to BFloat16 online.
 * Download the [Int8 pre-trained model](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/3dunet_int8_fully_quantized_perchannel.pb).
+* Download the [Int8 pre-trained model for OneDnn Graph](https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/3dunet_itex_int8.pb). Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point.
 
 In this example, we are using the model, trained using the fold 1 BRATS 2019 data.
 The validation files have been copied from [here](https://github.com/mlcommons/inference/tree/r0.7/vision/medical_imaging/3d-unet/folds).
@@ -80,7 +81,7 @@ Set the environment variables and one of the
 Set DATASET_DIR if you run `accuracy.sh` to calculate the model accuracy.
 
 ```
-# navigate to your Model Zoo directory
+# navigate to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset directory>
@@ -100,7 +101,7 @@ If not already setup, please follow instructions for [environment setup on Windo
 Set the environment variables and run `inference.sh` from the
 [quickstart script](#quick-start-scripts). Currently, for performance evaluation dummy data is used.
 ```
-# navigate to your Model Zoo directory
+# navigate to your AI Reference Models directory
 cd models
 
 set PRECISION=<set the precision "fp32" or "bfloat16">
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/bfloat16/model_init.py b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/bfloat16/model_init.py
index 8b842e8db..5d828d5ba 100644
--- a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/bfloat16/model_init.py
+++ b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/bfloat16/model_init.py
@@ -84,6 +84,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
             " --batch-size=" + str(self.args.batch_size) + \
             " --model-name=" + str(self.args.model_name)
 
+        if self.args.onednn_graph:
+            self.benchmark_command += "--onednn-graph"
+
         if not self.args.accuracy_only:
             self.benchmark_command += \
                 " --steps=" + str(self.args.steps) + " --warmup-steps=" + str(self.args.warmup_steps)
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/fp32/model_init.py b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/fp32/model_init.py
index 8b842e8db..5d828d5ba 100644
--- a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/fp32/model_init.py
+++ b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/fp32/model_init.py
@@ -84,6 +84,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
             " --batch-size=" + str(self.args.batch_size) + \
             " --model-name=" + str(self.args.model_name)
 
+        if self.args.onednn_graph:
+            self.benchmark_command += "--onednn-graph"
+
         if not self.args.accuracy_only:
             self.benchmark_command += \
                 " --steps=" + str(self.args.steps) + " --warmup-steps=" + str(self.args.warmup_steps)
diff --git a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/model_init.py b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/model_init.py
index fce1c2a77..9c32963f1 100644
--- a/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/model_init.py
+++ b/benchmarks/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/model_init.py
@@ -84,6 +84,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
             " --batch-size=" + str(self.args.batch_size) + \
             " --model-name=" + str(self.args.model_name)
 
+        if self.args.onednn_graph:
+            self.benchmark_command += "--onednn-graph"
+
         if not self.args.accuracy_only:
             self.benchmark_command += \
                 " --steps=" + str(self.args.steps) + " --warmup-steps=" + str(self.args.warmup_steps)
diff --git a/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 3a96ca3ba..000000000
--- a/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,58 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export MODEL_SRC_DIR=<path to the Mask RCNN models repo>
-```
-
-<model name> <precision> <mode> can be run for throughput and latency with `--batch-size=1`:
-
-```
-python launch_benchmark.py \
-  --model-source-dir ${MODEL_SRC_DIR} \
-  --model-name maskrcnn \
-  --framework tensorflow \
-  --precision fp32 \
-  --mode inference \
-  --batch-size 1 \
-  --socket-id 0 \
-  --data-location ${DATASET_DIR} \
-  --docker-image intel/intel-optimized-tensorflow:1.15.2 \
-  --output-dir ${OUTPUT_DIR}
-```
-
-Below is a sample log file tail when running benchmarking for throughput
-and latency:
-```
-Running per image evaluation...
-Evaluate annotation type *bbox*
-DONE (t=0.19s).
-Accumulating evaluation results...
-DONE (t=0.16s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.442
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.612
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.483
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.216
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.474
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.621
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.373
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.461
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.473
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.237
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.500
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.654
-Batch size: 1
-Time spent per BATCH: ... ms
-Total samples/sec: ... samples/s
-Total time:  ...
-Ran inference with batch size 1
-Log file location: {--output-dir value}/benchmark_maskrcnn_inference_fp32_20200917_164707.log
-```
diff --git a/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md b/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md
index 1c5228d96..b64dbf4f5 100644
--- a/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md
+++ b/benchmarks/image_segmentation/tensorflow/maskrcnn/inference/fp32/README.md
@@ -35,23 +35,23 @@ export DATASET_DIR=${DATASET_DIR}
 |-------------|-------------|
 | [fp32_inference.sh](fp32_inference.sh) | Runs inference with batch size 1 using Coco dataset and pretrained model|
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.6 or 3.7
         <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
@@ -70,7 +70,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pycocotools
         <li>scikit-image
         <li>scipy==1.2.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -101,7 +101,7 @@ the `MODEL_SRC_DIR` path set from the previous commands.
 Once the environment variables are all set, you can run the
 [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset>
diff --git a/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index c88167c85..000000000
--- a/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,41 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-pretrained model, model repository, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model>
-export TF_UNET_DIR=<path to the TF UNet directory tf_unet>
-```
-
-<model name> <precision> <mode> can be run to test batch and online inference using the
-following command:
-```
-python launch_benchmark.py \
-  --model-name unet \
-  --precision fp32 \
-  --mode inference \
-  --framework tensorflow \
-  --benchmark-only \
-  --batch-size 1 \
-  --socket-id 0 \
-  --checkpoint ${PRETRAINED_MODEL} \
-  --model-source-dir ${TF_UNET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --docker-image <docker image> \
-  -- checkpoint_name=model.ckpt
-```
-
-Below is an example of the log file tail:
-
-```
-Time spent per BATCH: ... ms
-Total samples/sec: ... samples/s
-Ran inference with batch size 1
-Log location outside container: {--output-dir value}/benchmark_unet_inference_fp32_20190201_205601.log
-```
diff --git a/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/README.md b/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/README.md
index eb0ee0fec..a55f161aa 100644
--- a/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/README.md
+++ b/benchmarks/image_segmentation/tensorflow/unet/inference/fp32/README.md
@@ -14,23 +14,23 @@ Intel Optimized TensorFlow.
 |-------------|-------------|
 | [fp32_inference.sh](fp32_inference.sh) | Runs inference with a batch size of 1 using a pretrained model |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
@@ -40,7 +40,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>click
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -74,7 +74,7 @@ the `TF_UNET_DIR` and `PRETRAINED_MODEL` paths set from the previous commands.
 Once the environment variables are all set, you can run a
 [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export OUTPUT_DIR=<path to the directory where log files will be written>
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/README.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/README.md
index e140a5ae8..fc13a1c2d 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/README.md
+++ b/benchmarks/language_modeling/tensorflow/bert_large/inference/README.md
@@ -39,17 +39,17 @@ Set the `DATASET_DIR` to point to that directory when running BERT Large inferen
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>unzip
@@ -59,7 +59,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
@@ -67,15 +67,15 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>numactl
         <li>unzip
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -96,6 +96,10 @@ Download the model pretrained frozen graph from the given link based on the prec
 * BFloat16:
   wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/optimized_bf16_bert.pb
 
+* Int8 for OneDNN Graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point):
+  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/bert_itex_int8.pb \
+  PRETRAINED_MODEL=$(pwd)/bert_itex_int8.pb
+
 ### Download checkpoints:
 ```bash
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/bert_large_checkpoints.zip
@@ -135,7 +139,7 @@ As an example, if the dataset location on Windows is `D:\user\wwm_uncased_L-24_H
 >Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/wwm_uncased_L-24_H-1024_A-16`.
 
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set the required environment vars: Only fp32 is supported on windows
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 3aca896a6..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,60 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export CHECKPOINT_DIR=<path to the pretrained model checkpoints>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* Benchmark
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR} \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    --benchmark-only \
-    -- infer_option=SQuAD
-  ```
-* Profile
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR}  \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    -- profile=True infer_option=SQuAD
-  ```
-* Accuracy
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR} \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    --accuracy-only \
-    -- infer_option=SQuAD
-  ```
-
-Output files and logs are saved to the ${OUTPUT_DIR} directory.
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/model_args.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/model_args.md
deleted file mode 100644
index d1ad1879a..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/.docs/advanced/model_args.md
+++ /dev/null
@@ -1,15 +0,0 @@
-<!-- 70. Model args -->
-Note that args specific to this model are specified after ` -- ` at
-the end of the command (like the `profile=True` arg in the Profile
-command above. Below is a list of all of the model specific args and
-their default values:
-
-| Model arg | Default value |
-|-----------|---------------|
-| doc_stride | `128` |
-| max_seq_length | `384` |
-| profile | `False` |
-| config_file | `bert_config.json` |
-| vocab_file | `vocab.txt` |
-| predict_file | `dev-v1.1.json` |
-| init_checkpoint | `model.ckpt-3649` |
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/model_init.py
index 7b45451f1..cd6e7c0cf 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/inference/bfloat16/model_init.py
@@ -159,6 +159,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.weight_sharing:
             model_args += " --weight_sharing"
 
+        if self.args.onednn_graph:
+            model_args += " --onednn_graph"
+
         model_args += " --num_cores_per_socket=" + str(platform_util.num_cores_per_socket)
 
         self.benchmark_command = self.get_command_prefix(args.socket_id) + \
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index ddd49e6d4..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,60 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export CHECKPOINT_DIR=<path to the pretrained model checkpoints>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* Benchmark
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR} \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    --benchmark-only \
-    -- infer_option=SQuAD
-  ```
-* Profile
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR}  \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    -- profile=True infer_option=SQuAD
-  ```
-* Accuracy
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp16 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location ${DATASET_DIR} \
-    --checkpoint ${CHECKPOINT_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image> \
-    --accuracy-only \
-    -- infer_option=SQuAD
-  ```
-
-Output files and logs are saved to the ${OUTPUT_DIR} directory.
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/model_args.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/model_args.md
deleted file mode 100644
index d1ad1879a..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/.docs/advanced/model_args.md
+++ /dev/null
@@ -1,15 +0,0 @@
-<!-- 70. Model args -->
-Note that args specific to this model are specified after ` -- ` at
-the end of the command (like the `profile=True` arg in the Profile
-command above. Below is a list of all of the model specific args and
-their default values:
-
-| Model arg | Default value |
-|-----------|---------------|
-| doc_stride | `128` |
-| max_seq_length | `384` |
-| profile | `False` |
-| config_file | `bert_config.json` |
-| vocab_file | `vocab.txt` |
-| predict_file | `dev-v1.1.json` |
-| init_checkpoint | `model.ckpt-3649` |
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/model_init.py
index bccf56e1e..d13702eba 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp16/model_init.py
@@ -168,6 +168,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.weight_sharing:
             model_args += " --weight_sharing"
 
+        if self.args.onednn_graph:
+            model_args += " --onednn_graph"
+
         if not self.args.gpu:
             model_args += " --num_cores_per_socket=" + str(platform_util.num_cores_per_socket)
             if self.args.warmup_steps:
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 8b92e929a..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,64 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, frozen graph, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export CHECKPOINT_DIR=<path to the pretrained model checkpoints>
-export PRETRAINED_MODEL=<path to the frozen graph .pb file>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* Benchmark
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=128 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    --benchmark-only \
-    -- infer_option=SQuAD
-  ```
-* Profile
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --docker-image <docker image> \
-    --output-dir $OUTPUT_DIR \
-    -- profile=True infer_option=SQuAD
-  ```
-* Accuracy
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --docker-image <docker image> \
-    --output-dir $OUTPUT_DIR \
-    --accuracy-only \
-    -- infer_option=SQuAD
-  ```
-
-Output files and logs are saved to the ${OUTPUT_DIR} directory.
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/model_args.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/model_args.md
deleted file mode 100644
index d1ad1879a..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/.docs/advanced/model_args.md
+++ /dev/null
@@ -1,15 +0,0 @@
-<!-- 70. Model args -->
-Note that args specific to this model are specified after ` -- ` at
-the end of the command (like the `profile=True` arg in the Profile
-command above. Below is a list of all of the model specific args and
-their default values:
-
-| Model arg | Default value |
-|-----------|---------------|
-| doc_stride | `128` |
-| max_seq_length | `384` |
-| profile | `False` |
-| config_file | `bert_config.json` |
-| vocab_file | `vocab.txt` |
-| predict_file | `dev-v1.1.json` |
-| init_checkpoint | `model.ckpt-3649` |
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/model_init.py
index 40caa2cd1..ce47dc7f1 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/inference/fp32/model_init.py
@@ -179,6 +179,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
             if self.args.weight_sharing:
                 model_args += " --weight_sharing"
 
+        if self.args.onednn_graph:
+            model_args += " --onednn_graph"
+
         self.benchmark_command = self.get_command_prefix(args.socket_id) + \
             self.python_exe + " " + model_script + model_args
 
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 61a17eede..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,65 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, frozen graph, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the squad dataset>
-export CHECKPOINT_DIR=<path to the pretrained model checkpoints>
-export PRETRAINED_MODEL=<path to the frozen graph .pb file>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* Benchmark
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=int8 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    --benchmark-only \
-    -- infer_option=SQuAD
-  ```
-* Profile
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=int8 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    --accuracy-only \
-    -- infer_option=SQuAD
-  ```
-* Accuracy
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=int8 \
-    --mode=inference \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --data-location $DATASET_DIR \
-    --checkpoint $CHECKPOINT_DIR \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    --accuracy-only \
-    -- infer_option=SQuAD
-  ```
-
-Output files and logs are saved to the ${OUTPUT_DIR} directory.
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/model_args.md b/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/model_args.md
deleted file mode 100644
index d1ad1879a..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/.docs/advanced/model_args.md
+++ /dev/null
@@ -1,15 +0,0 @@
-<!-- 70. Model args -->
-Note that args specific to this model are specified after ` -- ` at
-the end of the command (like the `profile=True` arg in the Profile
-command above. Below is a list of all of the model specific args and
-their default values:
-
-| Model arg | Default value |
-|-----------|---------------|
-| doc_stride | `128` |
-| max_seq_length | `384` |
-| profile | `False` |
-| config_file | `bert_config.json` |
-| vocab_file | `vocab.txt` |
-| predict_file | `dev-v1.1.json` |
-| init_checkpoint | `model.ckpt-3649` |
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/model_init.py
index 213eefb00..ab0c40f81 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/inference/int8/model_init.py
@@ -117,8 +117,12 @@ def __init__(self, args, custom_args=[], platform_util=None):
             else:
                 set_env_var("OMP_NUM_THREADS", platform_util.num_cores_per_socket)
 
-        model_script = os.path.join(
-            self.args.intelai_models, self.args.mode, "run_squad.py")
+        if self.args.onednn_graph:
+            model_script = os.path.join(
+                self.args.intelai_models, self.args.mode, "run_squad_int8.py")
+        else:
+            model_script = os.path.join(
+                self.args.intelai_models, self.args.mode, "run_squad.py")
 
         model_args = " --init_checkpoint=" + str(self.args.init_checkpoint) + \
                      " --vocab_file=" + str(self.args.vocab_file) + \
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/README.md b/benchmarks/language_modeling/tensorflow/bert_large/training/README.md
index 374c68124..86c4ecf38 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/README.md
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/README.md
@@ -70,16 +70,16 @@ in TensorFlow record file format. The output TensorFlow record files are expecte
 ### Setup on baremetal
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>numactl
         <li>unzip
@@ -95,7 +95,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li>intel-tensorflow>=2.5.0
@@ -107,7 +107,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>openssh-server (only required for multi-instance)
         <li>libopenmpi-dev (only required for multi-instance)
         <li>horovod==0.27.0 (only required for multi-instance)
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 9a436a985..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,243 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export CHECKPOINT_DIR=<path to the pretrained bert model directory>
-export DATASET_DIR=<path to GLUE data for classifier training or to the SQuAD data for SQuAD training>
-export OUTPUT_DIR=<directory where checkpoints and log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* **To run SQuAD training** use the following command with the `train_option=SQuAD`.  The
-  `CHECKPOINT_DIR` should point to the location where you've downloaded the BERT
-  large (whole word masking) pretrained model, and the `DATASET_DIR` should point to
-  the SQuAD data.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --docker-image <docker image> \
-    --output-dir $OUTPUT_DIR \
-    -- train_option=SQuAD \
-       vocab_file=$CHECKPOINT_DIR/vocab.txt \
-       config_file=$CHECKPOINT_DIR/bert_config.json \
-       init_checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       do_train=True \
-       train_file=$DATASET_DIR/train-v1.1.json \
-       do_predict=True \
-       predict_file=$DATASET_DIR/dev-v1.1.json \
-       learning_rate=3e-5 \
-       num_train_epochs=2 \
-       max_seq_length=384 \
-       doc_stride=128 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do_lower_case=True
-  ```
-  The dev set predictions will be saved to a file called predictions.json in the output directory.
-  ```
-  python ${DATASET_DIR}/evaluate-v1.1.py ${DATASET_DIR}/dev-v1.1.json ${OUTPUT_DIR}/predictions.json
-  ```
-  An execution with these parameters produces results in line with below scores:
-  ```
-  Bf16: {"exact_match": 86.77388836329234, "f1": 92.98642358746287}
-  FP32: {"exact_match": 86.72658467360453, "f1": 92.98046893150796}
-  ```
-
-  To run distributed training of SQuAD: for better throughput, simply specify `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`.
-  To set `--mpi_num_processes=<num_of_sockets>`, please run `lscpu` on your machine to check the available number of sockets.
-  >Note:
-  >- the `global batch size` is `mpi_num_processes * train_batch_size` and sometimes `learning rate` needs to be adjusted for convergence.
-  >- `square root learning rate scaling` is used by default.
-  >- for BERT fine-tuning, state-of-the-art accuracy can be achieved via parallel training without synchronizing gradients between MPI workers.
-  >- The `--mpi_workers_sync_gradients=[True/False]` controls whether the MPI workers sync gradients.By default it is set to `False` meaning the workers are training independently and the best performing training results will be picked in the end.
-  >- To enable gradients synchronization, set the `--mpi_workers_sync_gradients` to `True` in BERT-specific options.
-  >- The options `optimized_softmax=True` can be set for better performance.
-
-  Note : Add a space after ```--```, for BERT-specific options.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --mpi_num_processes=<num_of_sockets> \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train_option=SQuAD \
-       vocab_file=$CHECKPOINT_DIR/vocab.txt \
-       config_file=$CHECKPOINT_DIR/bert_config.json \
-       init_checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       do_train=True \
-       train_file=$DATASET_DIR/train-v1.1.json \
-       do_predict=True \
-       predict_file=$DATASET_DIR/dev-v1.1.json \
-       learning_rate=3e-5 \
-       num_train_epochs=2 \
-       max_seq_length=384 \
-       doc_stride=128 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do_lower_case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-
-* **To run Classifier** training, use the following command with the
-  `train-option=Classifier`. For classifier training, use the GLUE data in
-  as the `DATASET_DIR`. The `CHECKPOINT_DIR` should point to the BERT base
-  uncased 12-layer, 768-hidden pretrained model that you've downloaded.
-  This example code fine-tunes BERT-Base on the Microsoft Research Paraphrase
-  Corpus (MRPC) corpus, which only contains 3,600 examples.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Classifier \
-       task-name=MRPC \
-       do-train=true \
-       do-eval=true \
-       data-dir=$DATASET_DIR/MRPC \
-       vocab-file=$CHECKPOINT_DIR/vocab.txt \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=128 \
-       learning-rate=2e-5 \
-       num-train-epochs=30 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do-lower-case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-
-  To run distributed training of Classifier: for better throughput, specify
-  `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`,
-  please run `lscpu` on your machine to check the available number of sockets.
-  Note that the `global batch size` is `mpi_num_processes * train_batch_size`
-  and sometimes learning rate needs to be adjusted for convergence. By default,
-  the script uses square root learning rate scaling.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --mpi_num_processes=<num_of_sockets> \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Classifier \
-       task-name=MRPC \
-       do-train=true \
-       do-eval=true \
-       data-dir=$DATASET_DIR/MRPC \
-       vocab-file=$CHECKPOINT_DIR/vocab.txt \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=128 \
-       learning-rate=2e-5 \
-       num-train-epochs=30 \
-       optimized_softmax=True \
-       experimental_gelu=True \
-       do-lower-case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-* **To run pre-training from scratch**, use the following command with
-  `train-option=Pretraining`. Pre-training has two phases:
-  In the first phase, the data is generated for sequential length 128.
-  And in the second phase, sequential length 512 is used.
-  Please follow instructions in [google bert pre-training](https://github.com/google-research/bert#pre-training-with-bert)
-  for data pre-processing and set the `DATASET_DIR` environment variable
-  to the TF record directory. The `CHECKPOINT_DIR` should point to the
-  BERT large uncased (whole word masking) directory.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --socket-id=0 \
-    --num-intra-threads=24 \
-    --num-inter-threads=1 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Pretraining \
-       input-file=$DATASET_DIR/tf_wiki_formatted_slen512.tfrecord \
-       do-train=True \
-       do-eval=True \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=512 \
-       max-predictions=76 \
-       num-train-steps=20 \
-       warmup-steps=10 \
-       learning-rate=2e-5 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       profile=False
-  ```
-
-  To run distributed training of pretraining: for better throughput, simply specify
-  `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`. Please
-  run `lscpu` on your machine to check the available number of sockets.
-  > Note that the `global batch size` is `mpi_num_processes * train_batch_size` and
-  > sometimes `learning rate` needs to be adjusted for convergence. By default, the
-  > script uses `square root learning rate scaling`.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=bfloat16 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --num-intra-threads=22 \
-    --num-inter-threads=1 \
-    --mpi_num_processes=4 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Pretraining \
-       input-file=$DATASET_DIR/tf_wiki_formatted_slen512.tfrecord \
-       do-train=True \
-       do-eval=True \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=512 \
-       max-predictions=76 \
-       num-train-steps=20 \
-       warmup-steps=10 \
-       learning-rate=2e-5 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       profile=False
-  ```
-
-  >Note: for best performance, we will set `num-intra-thread` as follows:
-  >- For single instance (mpi_num_processes=1) run: the value is equal to
-  > number of logical cores per socket.
-  >- For multi-instance (mpi_num_processes>1) run: the value is equal to
-  > (num_of_logical_cores_per_socket - 2)
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
index a478423a9..716288f4b 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
@@ -156,7 +156,8 @@ def __init__(self, args, custom_args=[], platform_util=None):
             " --do_lower_case=" + str(self.args.do_lower_case) + eoo + \
             " --experimental_gelu=" + str(self.args.experimental_gelu) + eoo + \
             " --optimized_softmax=" + str(self.args.optimized_softmax) + eoo + \
-            " --mpi_workers_sync_gradients=" + str(self.args.mpi_workers_sync_gradients) + eoo
+            " --mpi_workers_sync_gradients=" + str(self.args.mpi_workers_sync_gradients) + eoo + \
+            " --gpu=" + str(self.args.gpu) + eoo
 
         if self.args.train_option == "SQuAD":
             self.cmd_args = self.cmd_args + \
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 73c4b9141..000000000
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,243 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the directory, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export CHECKPOINT_DIR=<path to the pretrained bert model directory>
-export DATASET_DIR=<path to GLUE data for classifier training or to the SQuAD data for SQuAD training>
-export OUTPUT_DIR=<directory where checkpoints and log files will be saved>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* **To run SQuAD training** use the following command with the `train_option=SQuAD`. The
-  `CHECKPOINT_DIR` should point to the location where you've downloaded the BERT
-  large (whole word masking) pretrained model, and the `DATASET_DIR` should point to
-  the SQuAD data.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train_option=SQuAD \
-       vocab_file=$CHECKPOINT_DIR/vocab.txt \
-       config_file=$CHECKPOINT_DIR/bert_config.json \
-       init_checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       do_train=True \
-       train_file=$DATASET_DIR/train-v1.1.json \
-       do_predict=True \
-       predict_file=$DATASET_DIR/dev-v1.1.json \
-       learning_rate=3e-5 \
-       num_train_epochs=2 \
-       max_seq_length=384 \
-       doc_stride=128 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do_lower_case=True
-  ```
-  The dev set predictions will be saved to a file called predictions.json in the output directory.
-  ```
-  python ${DATASET_DIR}/evaluate-v1.1.py ${DATASET_DIR}/dev-v1.1.json ${OUTPUT_DIR}/predictions.json
-  ```
-  An execution with these parameters produces results in line with below scores:
-  ```
-  Bf16: {"exact_match": 86.77388836329234, "f1": 92.98642358746287}
-  FP32: {"exact_match": 86.72658467360453, "f1": 92.98046893150796}
-  ```
-
-  To run distributed training of SQuAD: for better throughput, simply specify `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`.
-  To set `--mpi_num_processes=<num_of_sockets>`, please run `lscpu` on your machine to check the available number of sockets.
-  >Note:
-  >- the `global batch size` is `mpi_num_processes * train_batch_size` and sometimes `learning rate` needs to be adjusted for convergence.
-  >- `square root learning rate scaling` is used by default.
-  >- for BERT fine-tuning, state-of-the-art accuracy can be achieved via parallel training without synchronizing gradients between MPI workers.
-  >- The `--mpi_workers_sync_gradients=[True/False]` controls whether the MPI workers sync gradients.By default it is set to `False` meaning the workers are training independently and the best performing training results will be picked in the end.
-  >- To enable gradients synchronization, set the `--mpi_workers_sync_gradients` to `True` in BERT-specific options.
-  >- The options `optimized_softmax=True` can be set for better performance.
-
-  Note : Add a space after ```--```, for BERT-specific options.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --mpi_num_processes=<num_of_sockets> \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train_option=SQuAD \
-       vocab_file=$CHECKPOINT_DIR/vocab.txt \
-       config_file=$CHECKPOINT_DIR/bert_config.json \
-       init_checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       do_train=True \
-       train_file=$DATASET_DIR/train-v1.1.json \
-       do_predict=True \
-       predict_file=$DATASET_DIR/dev-v1.1.json \
-       learning_rate=3e-5 \
-       num_train_epochs=2 \
-       max_seq_length=384 \
-       doc_stride=128 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do_lower_case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-
-* **To run Classifier** training, use the following command with the
-  `train-option=Classifier`. For classifier training, use the GLUE data in
-  as the `DATASET_DIR`. The `CHECKPOINT_DIR` should point to the BERT base
-  uncased 12-layer, 768-hidden pretrained model that you've downloaded.
-  This example code fine-tunes BERT-Base on the Microsoft Research Paraphrase
-  Corpus (MRPC) corpus, which only contains 3,600 examples.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Classifier \
-       task-name=MRPC \
-       do-train=true \
-       do-eval=true \
-       data-dir=$DATASET_DIR/MRPC \
-       vocab-file=$CHECKPOINT_DIR/vocab.txt \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=128 \
-       learning-rate=2e-5 \
-       num-train-epochs=30 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       do-lower-case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-
-  To run distributed training of Classifier: for better throughput, specify
-  `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`,
-  please run `lscpu` on your machine to check the available number of sockets.
-  Note that the `global batch size` is `mpi_num_processes * train_batch_size`
-  and sometimes learning rate needs to be adjusted for convergence. By default,
-  the script uses square root learning rate scaling.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=32 \
-    --mpi_num_processes=<num_of_sockets> \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Classifier \
-       task-name=MRPC \
-       do-train=true \
-       do-eval=true \
-       data-dir=$DATASET_DIR/MRPC \
-       vocab-file=$CHECKPOINT_DIR/vocab.txt \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=128 \
-       learning-rate=2e-5 \
-       num-train-epochs=30 \
-       optimized_softmax=True \
-       experimental_gelu=True \
-       do-lower-case=True
-  ```
-  The results file will be written to the `${OUTPUT_DIR}` directory.
-* **To run pre-training from scratch**, use the following command with
-  `train-option=Pretraining`. Pre-training has two phases:
-  In the first phase, the data is generated for sequential length 128.
-  And in the second phase, sequential length 512 is used.
-  Please follow instructions in [google bert pre-training](https://github.com/google-research/bert#pre-training-with-bert)
-  for data pre-processing and set the `DATASET_DIR` environment variable
-  to the TF record directory. The `CHECKPOINT_DIR` should point to the
-  BERT large uncased (whole word masking) directory.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --socket-id=0 \
-    --num-intra-threads=24 \
-    --num-inter-threads=1 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Pretraining \
-       input-file=$DATASET_DIR/tf_wiki_formatted_slen512.tfrecord \
-       do-train=True \
-       do-eval=True \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=512 \
-       max-predictions=76 \
-       num-train-steps=20 \
-       warmup-steps=10 \
-       learning-rate=2e-5 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       profile=False
-  ```
-
-  To run distributed training of pretraining: for better throughput, simply specify
-  `--mpi_num_processes=<num_of_sockets> [--mpi_num_processes_per_socket=1]`. Please
-  run `lscpu` on your machine to check the available number of sockets.
-  > Note that the `global batch size` is `mpi_num_processes * train_batch_size` and
-  > sometimes `learning rate` needs to be adjusted for convergence. By default, the
-  > script uses `square root learning rate scaling`.
-  ```
-  python launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=fp32 \
-    --mode=training \
-    --framework=tensorflow \
-    --batch-size=24 \
-    --num-intra-threads=22 \
-    --num-inter-threads=1 \
-    --mpi_num_processes=4 \
-    --volume $CHECKPOINT_DIR:$CHECKPOINT_DIR \
-    --volume $DATASET_DIR:$DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image> \
-    -- train-option=Pretraining \
-       input-file=$DATASET_DIR/tf_wiki_formatted_slen512.tfrecord \
-       do-train=True \
-       do-eval=True \
-       config-file=$CHECKPOINT_DIR/bert_config.json \
-       init-checkpoint=$CHECKPOINT_DIR/bert_model.ckpt \
-       max-seq-length=512 \
-       max-predictions=76 \
-       num-train-steps=20 \
-       warmup-steps=10 \
-       learning-rate=2e-5 \
-       optimized_softmax=True \
-       experimental_gelu=False \
-       profile=False
-  ```
-
-  >Note: for best performance, we will set `num-intra-thread` as follows:
-  >- For single instance (mpi_num_processes=1) run: the value is equal to
-  > number of logical cores per socket.
-  >- For multi-instance (mpi_num_processes>1) run: the value is equal to
-  > (num_of_logical_cores_per_socket - 2)
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
index 9f1f8d521..eaf002b36 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
@@ -158,7 +158,8 @@ def __init__(self, args, custom_args=[], platform_util=None):
             " --profile=" + str(self.args.profile) + eoo + \
             " --do_lower_case=" + str(self.args.do_lower_case) + eoo + \
             " --experimental_gelu=" + str(self.args.experimental_gelu) + eoo + \
-            " --mpi_workers_sync_gradients=" + str(self.args.mpi_workers_sync_gradients) + eoo
+            " --mpi_workers_sync_gradients=" + str(self.args.mpi_workers_sync_gradients) + eoo + \
+            " --gpu=" + str(self.args.gpu) + eoo
 
         if self.args.train_option == "SQuAD":
             self.cmd_args = self.cmd_args + \
diff --git a/benchmarks/language_modeling/tensorflow/distilbert_base/inference/README.md b/benchmarks/language_modeling/tensorflow/distilbert_base/inference/README.md
index 368ee6b42..aeecd24b8 100644
--- a/benchmarks/language_modeling/tensorflow/distilbert_base/inference/README.md
+++ b/benchmarks/language_modeling/tensorflow/distilbert_base/inference/README.md
@@ -17,6 +17,9 @@ wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/dis
 Download Frozen graph for INT8:
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/FusedBMM_int8_distilbert_frozen_graph.pb
 
+Download Frozen graph for IN8 with OneDNN graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point):
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/distilbert_itex_int8.pb
+
 Downloading dataset:
 cd to directory:  <model_zoo_dir>/models/language_modeling/tensorflow/distilbert_base/inference/
 python download_dataset.py --path_to_save_dataset <enter path to save dataset>
diff --git a/benchmarks/language_translation/tensorflow/bert/inference/README.md b/benchmarks/language_translation/tensorflow/bert/inference/README.md
index 879bd19ad..2fb22d1f6 100644
--- a/benchmarks/language_translation/tensorflow/bert/inference/README.md
+++ b/benchmarks/language_translation/tensorflow/bert/inference/README.md
@@ -28,8 +28,6 @@ You can also use the helper script [download_glue_data.py](https://gist.github.c
 | [`accuracy.sh`](/quickstart/language_translation/tensorflow/bert/inference/cpu/accuracy.sh) | Runs accuracy |
 
 
-
-<!--- 50. AI Kit -->
 ## Run the model
 * Download the fp32 "BERT-Base, Uncased" pre-trained model and unzip it inside the dataset directory `data/MRPC`.
 If you run on Windows, please use a browser to download the pretrained model using the link below. For Linux, run:
@@ -52,7 +50,7 @@ If you run on Windows, please use a browser to download the pretrained model usi
   If you re-run multiple times (making sure to point to different `output-dir`), you should see results between 84% and 88%.
 
 
-* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [AI Reference Models Github repository](https://github.com/IntelAI/models).
   ```
   git clone https://github.com/IntelAI/models.git
   ```
@@ -62,7 +60,7 @@ If you run on Windows, please use a browser to download the pretrained model usi
 
 ## Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
@@ -106,4 +104,3 @@ As an example, if the pretrained model path on Windows is `D:\user\transformer_l
 
 * To run more advanced use cases, see the instructions [here](fp32/Advanced.md)
   for calling the `launch_benchmark.py` script directly.
-
diff --git a/benchmarks/language_translation/tensorflow/bert/inference/fp32/README.md b/benchmarks/language_translation/tensorflow/bert/inference/fp32/README.md
deleted file mode 100644
index 9814010ef..000000000
--- a/benchmarks/language_translation/tensorflow/bert/inference/fp32/README.md
+++ /dev/null
@@ -1,106 +0,0 @@
-<!--- 0. Title -->
-# BERT FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running BERT FP32 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-### Dataset
-Download Microsoft Research Paraphrase Corpus (MRPC) data in cloned repository and save it inside `data` folder.
-You can also use the helper script [download_glue_data.py](https://gist.github.com/vlasenkoalexey/fef1601580f269eca73bf26a198595f3) to download the data:
-
-   ```
-   # Obtain a copy of download_glue_data.py to the current directory
-   wget https://gist.githubusercontent.com/vlasenkoalexey/fef1601580f269eca73bf26a198595f3/raw/db67cdf22eb5bd7efe376205e8a95028942e263d/download_glue_data.py
-   python3 download_glue_data.py --data_dir ./data/ --tasks MRPC
-   ```
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`inference_realtime.sh`](/quickstart/language_translation/tensorflow/bert/inference/cpu/inference_realtime.sh) | Runs online inference (batch_size=1). |
-| [`inference_throughput.sh`](/quickstart/language_translation/tensorflow/bert/inference/cpu/inference_throughput.sh) | Runs batch inference (batch_size=32). |
-| [`accuracy.sh`](/quickstart/language_translation/tensorflow/bert/inference/cpu/accuracy.sh) | Runs accuracy |
-
-
-<!--- 50. AI Kit -->
-## Run the model
-* Download the fp32 "BERT-Base, Uncased" pre-trained model and unzip it inside the dataset directory `data/MRPC`.
-If you run on Windows, please use a browser to download the pretrained model using the link below. For Linux, run:
-
-   ```
-   cd data/MRPC
-   wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
-   unzip uncased_L-12_H-768_A-12.zip
-   cd ../..
-   ```
-
-* Clone [google-research/bert](https://github.com/google-research/bert) repository:
-   ```
-   git clone --single-branch https://github.com/google-research/bert.git && cd bert/
-   git checkout 88a817c37f788702a363ff935fd173b6dc6ac0d6
-   ```
-   Set the `MODEL_SOURCE` environment variable to the location of `bert` directory.
-   
-* Small sets like MRPC have a high variance in the Dev set accuracy, even when starting from the same pre-training checkpoint.
-  If you re-run multiple times (making sure to point to different `output-dir`), you should see results between 84% and 88%.
-
-
-* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
-  ```
-  git clone https://github.com/IntelAI/models.git
-  ```
-
-* Install [intel-tensorflow>=2.5.0](https://pypi.org/project/intel-tensorflow/) on your system.
-
-
-## Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-# Set env vars
-export DATASET_DIR=<path to the MRPC directory>           
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model which is downloaded under MRPC directory>
-export MODEL_SOURCE=<path to the bert directory>
-
-# Run a quickstart script
-./quickstart/language_translation/tensorflow/bert/inference/cpu/<script name>.sh
-```
-
-## Run on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-cd models
-
-# Set environment variables
-set DATASET_DIR=<path to the MRPC directory>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-set PRETRAINED_MODEL=<path to the pretrained model which is downloaded under MRPC directory>
-set MODEL_SOURCE=<path to the bert directory>
-
-# Run a quickstart script
-bash quickstart\language_translation\tensorflow\bert\inference\cpu\<script name>.sh
-```
-
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the pretrained model path on Windows is `D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb
-> /d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb
->```
->Then, set the `PRETRAINED_MODEL` environment variable `set PRETRAINED_MODEL=/d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb`.
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-To run more advanced use cases, see the instructions [here](Advanced.md)
-for calling the `launch_benchmark.py` script directly.
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md
index bab4f60f0..e94a8fe2b 100644
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md
+++ b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md
@@ -29,20 +29,20 @@ Set the `DATASET_DIR` to point as instructed above  when running MLPerf GNMT.
 | [`batch_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=32). |
 | [`accuracy.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh) | Runs accuracy |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>git
         <li>numactl
@@ -54,7 +54,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
@@ -63,7 +63,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pip
         <li>wget
         <li><a href="https://bazel.build/">Bazel</a> to build tensorflow addons
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -83,7 +83,7 @@ then build and install the TensorFlow addons wheel.
 ```
 # TensorFlow addons (r0.5) build and installation instructions:
 #   Clone TensorFlow addons (r0.5) and apply a patch: A patch file
-#   is attached in Intel Model Zoo MLpref GNMT model scripts,
+#   is attached in Intel AI Reference Models MLpref GNMT model scripts,
 #   it fixes TensorFlow addons (r0.5) to work with TensorFlow
 #   version 2.11, and prevents TensorFlow 2.0.0 to be installed
 #   by default as a required dependency.
@@ -104,7 +104,7 @@ Once that has been completed, ensure you have the required environment variables
 set, and then run a quickstart script.
 
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env var paths
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index a5f0abc7a..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,65 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-If you are going to run using docker, copy the `tensorflow-addons` wheel that you built
-during the [model setup](README.md#run-the-model) to the model zoo's `mlperf_gnmt` directory:
-```
-cp <tensorflow-addons repo>/artifacts/tensorflow_addons-*.whl <model zoo directory>/models/language_translation/tensorflow/mlperf_gnmt
-```
-
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph .pb file>
-```
-
-<model name> <mode> can be run in three different modes:
-
-* For online inference, use the following command (with `--benchmark-only`, `--socket-id 0` and `--batch-size 1`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 1 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --benchmark-only
-  ```
-* For batch inference, use the following command (with `--benchmark-only`, `--socket-id 0` and `--batch-size 32`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 32 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --benchmark-only
-  ```
-* For accuracy testing, use the following command (with `--accuracy_only`, `--socket-id 0` and `--batch-size 32`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 32 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --accuracy-only
-  ```
diff --git a/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/README.md b/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/README.md
index 80029966c..c7fc0f54f 100644
--- a/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/README.md
+++ b/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/README.md
@@ -35,21 +35,21 @@ The path to the directory with these files should be set as the
 | [`online_inference.sh`](/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
 | [`batch_inference.sh`](/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=64). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>Cython
         <li>numactl
@@ -60,7 +60,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li>intel-tensorflow>=2.5.0
@@ -69,17 +69,17 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>numactl
         <li>pandas
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
         <li>Cython
         <li>pandas
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -95,12 +95,12 @@ tar -xzf transformer_lt_official_fp32_pretrained_model.tar.gz
 export PRETRAINED_MODEL=$(pwd)/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb
 ```
 
-Navigate to the model zoo directory, ensure that you have all the required
+Navigate to the AI Reference Models directory, ensure that you have all the required
 environment variables set, then run a [quickstart script](#quick-start-scripts) on either [Linux](#run-on-linux) or [Windows](#run-on-windows) systems.
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
@@ -118,7 +118,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
diff --git a/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 2b85922a9..000000000
--- a/benchmarks/language_translation/tensorflow/transformer_lt_official/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,58 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph .pb file>
-```
-
-Transformer LT official can run for online or batch inference. Use one of the
-following examples below, depending on your use case.
-
-For online inference (using `--socket-id 0` and `--batch-size 1`):
-```
-python launch_benchmark.py \
-    --model-name transformer_lt_official \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --batch-size 1 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    -- file=newstest2014.en \
-    file_out=translate.txt \
-    reference=newstest2014.de \
-    vocab_file=vocab.txt
-```
-
-For batch inference (using `--socket-id 0` and `--batch-size 64`):
-```
-python launch_benchmark.py \
-    --model-name transformer_lt_official \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --batch-size 64 \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR} \
-    -- file=newstest2014.en \
-    file_out=translate.txt \
-    reference=newstest2014.de \
-    vocab_file=vocab.txt
-
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-The num-inter-threads and num-intra-threads could be set different numbers
-depending on the CPU in the system to achieve the best performance.
diff --git a/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/README.md b/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/README.md
index 84743c5b6..156361298 100644
--- a/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/README.md
+++ b/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/README.md
@@ -23,12 +23,11 @@ Set `DATASET_DIR` to point out to the location of the dataset directory.
 | `inference_throughput_multi_instance.sh` | Runs multi instance batch inference with batch-size=448 for precisions (int8, fp32, bfloat16) to get throughput using 1 instance per socket. |
 | `accuracy.sh` | Measures the inference accuracy for the specified precision (int8, fp32, or bfloat16). |
 
-<!--- 50. AI Kit -->
 ## Run the model
 
 Setup your environment using the instructions below,
 
-* Follow the instructions to setup your bare metal environment on Linux system. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
+* Follow the instructions to setup your bare metal environment on Linux system. Ensure that you have a clone of the [AI Reference Models Github repository](https://github.com/IntelAI/models).
   ```
   git clone https://github.com/IntelAI/models.git
   ```
@@ -39,7 +38,7 @@ Setup your environment using the instructions below,
 
   While using Int8 precision:
 
-  Install [intel-tensorflow==2.11.dev202242](https://pypi.org/project/intel-tensorflow/2.11.dev202242/) on your system. 
+  Install [intel-tensorflow==2.11.dev202242](https://pypi.org/project/intel-tensorflow/2.11.dev202242/) on your system.
 
 After installing the prerequisites, download the pretrained model and set
 the `PRETRAINED_MODEL` environment variable to the .pb file path:
@@ -48,13 +47,17 @@ the `PRETRAINED_MODEL` environment variable to the .pb file path:
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_int8.pb
 export PRETRAINED_MODEL=$(pwd)/transformer_mlperf_int8.pb
 
+# Int8 Pretrained model for OneDNN Graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point):
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/transformer_lt_itex_int8.pb
+export PRETRAINED_MODEL=$(pwd)/transformer_lt_itex_int8.pb
+
 # FP32 Pretrained model:
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_fp32.pb
 export PRETRAINED_MODEL=$(pwd)/transformer_mlperf_fp32.pb
 
 # BFloat16 Pretrained model:
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_bf16.pb
-export PRETRAINED_MODEL=$(pwd)/transformer_mlperf_bfloat16.pb
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/transformer_bf16_frozen_8_30.pb
+export PRETRAINED_MODEL=$(pwd)/transformer_bf16_frozen_8_30.pb
 ```
 
 Once that has been completed, ensure you have the required environment variables
@@ -62,7 +65,7 @@ set, and then run a quickstart script.
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set the required environment vars
@@ -80,7 +83,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
diff --git a/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/int8/model_init.py b/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/int8/model_init.py
index 5d36cc8f4..a41a03863 100644
--- a/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/int8/model_init.py
+++ b/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/int8/model_init.py
@@ -107,6 +107,9 @@ def __init__(self, args, custom_args, platform_util=None):
                    " --inter_op_parallelism_threads=" + str(self.args.num_inter_threads) + \
                    " --intra_op_parallelism_threads=" + str(self.args.num_intra_threads)
 
+        if self.args.onednn_graph:
+            cmd_args += " --onednn-graph"
+
         self.bleu_params += " --translation=" + translate_file + \
                             " --reference=" + self.args.reference
 
diff --git a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/README.md b/benchmarks/language_translation/tensorflow/transformer_mlperf/training/README.md
index a4f9bfbb8..5faf545e8 100644
--- a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/README.md
+++ b/benchmarks/language_translation/tensorflow/transformer_mlperf/training/README.md
@@ -21,20 +21,20 @@ Set `DATASET_DIR` to point out to the location of the dataset directory.
 | [`training_demo.sh`](/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training_demo.sh) | Runs 100 training steps. The script runs in single instance mode by default, for multi instance mode set `MPI_NUM_PROCESSES`. |
 | [`training.sh`](/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh) | Uses mpirun to execute 2 processes with 1 process per socket with a batch size of 5120 for the specified precision (fp32 or bfloat16). Logs for each instance are saved to the output directory. |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>numactl
         <li>gcc-8 (for multi-instance only)
@@ -48,7 +48,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li>intel-tensorflow>=2.5.0
@@ -60,7 +60,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>openmpi (for multi-instance only)
         <li>openssh (for multi-instance only)
         <li>horovod (for multi-instance only)
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -69,11 +69,11 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
 
 After you've completed the setup, export environment variables for the `DATASET_DIR`
 and an `OUTPUT_DIR` where log files will be written, then run a
-[quickstart script](#quick-start-scripts) from your model zoo directory. To run
+[quickstart script](#quick-start-scripts) from your AI Reference Models directory. To run
 multi-instance training, set the `MPI_NUM_PROCESSES` var to specify the number of
 instances to run (for example `export MPI_NUM_PROCESSES=2` for 2 instances).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set required environment variables
diff --git a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_translation/tensorflow/transformer_mlperf/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index deae76b54..000000000
--- a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,176 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files and checkpoints will be written>
-```
-
-Transformer Language in mlperf benchmark can run with full training or
-fewer training steps. During training we can control if it will do the evaluation
-or not.
-
-Note user can specify the batch size suitable to their system with command option `--batch-size` to achieve the best performance. If users don't specify the batch size, then the model will choose default batch size, which is set to 5120.
-
-For training with all epochs, with evaluation, and with saving checkpoints (for convergence):
-```
-python launch_benchmark.py \
-    --framework tensorflow  \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=0 steps_between_eval=0 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50
-```
-The Tensorflow binary in the docker image needed to be compiled correctly so that Bfloat16 code is included.
-
-For training with fewer training steps, such as 200 steps, and with no evaluation:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="Yes" do_eval="No" print_iter=50
-```
-
-For training with fewer training steps, and with evaluation:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50 \
-    bleu_source=/home/<user>/newstest2014.en --bleu_ref=/home/<user>/newstest2014.de
-```
-
-For training only for Benchmarking, to reduce training time,
-"saving checkpoints" and "doing the evaluation" can be disabled as below:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="No" do_eval="No" print_iter=50
-```
-
-All the above commands run on a single socket of the CPU and calculate the number of cores
-using `BaseModelInitializer.set_num_inter_intra_threads()`.
-
-For training in multi-instance mode (2 sockets in a single node for example) in evaluation mode,
-where you are "saving checkpoints" and "doing the evaluation":
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --mpi_num_processes=2 \
-    --num-intra-threads=26 --num-inter-threads=1 \
-    -- random_seed=11 train_steps=0 steps_between_eval=0 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50
-```
-
-For training only in multi-instance mode (4 sockets in a single node for example) for benchmarking,
-"saving checkpoints" and "doing the evaluation" can be disabled as below:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --mpi_num_processes=4 \
-    --num-intra-threads=26 --num-inter-threads=1 \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="No" do_eval="No" print_iter=50
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output. The above two command assumes 2 sockets in the CPU,
-and 28 cores per socket. We keep aside 2 cores for horovod communications. Thus,
-in general the num-intra-threads is equal to "available cores per socket - 2".
-
-The transformer model related parameters is appended after "-- "
-the num-intra-threads is equal to available cores - 2.
-
-The log file is saved to the `$OUTPUT_DIR` directory. Below are
-examples of what the tail of your log file should look like for the
-different configs.
-
-Example log tail when running training without saving checkpoints and not doing the evaluations:
-```
-INFO:tensorflow:loss = 9.804764, step = 0
-I0430 15:43:41.658226 140067518572352 basic_session_run_hooks.py:262] loss = 9.804764, step = 0
-INFO:tensorflow:loss = 9.804764
-I0430 15:43:41.658570 140067518572352 basic_session_run_hooks.py:262] loss = 9.804764
-INFO:tensorflow:Batch [10]:  last 10 steps exp/sec = 1777.54, completed 10/50 warmup steps
-I0430 15:44:10.461924 140067518572352 hooks.py:117] Batch [10]:  last 10 steps exp/sec = 1777.54, completed 10/50 warmup steps
-INFO:tensorflow:loss = 9.741728 (28.804 sec)
-I0430 15:44:10.462378 140067518572352 basic_session_run_hooks.py:260] loss = 9.741728 (28.804 sec)
-INFO:tensorflow:Batch [20]:  last 10 steps exp/sec = 1910.88, completed 20/50 warmup steps
-I0430 15:44:37.255764 140067518572352 hooks.py:117] Batch [20]:  last 10 steps exp/sec = 1910.88, completed 20/50 warmup steps
-INFO:tensorflow:loss = 9.638916 (26.794 sec)
-I0430 15:44:37.255995 140067518572352 basic_session_run_hooks.py:260] loss = 9.638916 (26.794 sec)
-INFO:tensorflow:Batch [30]:  last 10 steps exp/sec = 1898.28, completed 30/50 warmup steps
-I0430 15:45:04.227502 140067518572352 hooks.py:117] Batch [30]:  last 10 steps exp/sec = 1898.28, completed 30/50 warmup steps
-INFO:tensorflow:loss = 9.4281845 (26.972 sec)
-I0430 15:45:04.227731 140067518572352 basic_session_run_hooks.py:260] loss = 9.4281845 (26.972 sec)
-INFO:tensorflow:Batch [40]:  last 10 steps exp/sec = 1869.21, completed 40/50 warmup steps
-I0430 15:45:31.618821 140067518572352 hooks.py:117] Batch [40]:  last 10 steps exp/sec = 1869.21, completed 40/50 warmup steps
-INFO:tensorflow:loss = 9.169721 (27.392 sec)
-I0430 15:45:31.619261 140067518572352 basic_session_run_hooks.py:260] loss = 9.169721 (27.392 sec)
-INFO:tensorflow:Batch [50]:  last 10 steps exp/sec = 1943.57, completed 50/50 warmup steps
-I0430 15:45:57.962067 140067518572352 hooks.py:117] Batch [50]:  last 10 steps exp/sec = 1943.57, completed 50/50 warmup steps
-```
-
-Multi-instance sample log:
-Running 2 instances only. Both instances printing loss values.
-```
-INFO:tensorflow:loss = 9.881177, step = 0
-I0430 14:29:19.457713 140147178080064 basic_session_run_hooks.py:262] loss = 9.881177, step = 0
-INFO:tensorflow:loss = 9.881177
-I0430 14:29:19.458147 140147178080064 basic_session_run_hooks.py:262] loss = 9.881177
-INFO:tensorflow:loss = 9.916411, step = 0
-I0430 14:29:19.486985 140684161857344 basic_session_run_hooks.py:262] loss = 9.916411, step = 0
-INFO:tensorflow:loss = 9.916411
-I0430 14:29:19.487454 140684161857344 basic_session_run_hooks.py:262] loss = 9.916411
-INFO:tensorflow:Batch [10]:  last 10 steps exp/sec = 1368.86, completed 10/50 warmup steps
-I0430 14:29:56.861132 140147178080064 hooks.py:117] Batch [10]:  last 10 steps exp/sec = 1368.86, completed 10/50 warmup steps
-INFO:tensorflow:loss = 9.821904 (37.403 sec)
-I0430 14:29:56.861575 140147178080064 basic_session_run_hooks.py:260] loss = 9.821904 (37.403 sec)
-INFO:tensorflow:Batch [10]:  last 10 steps exp/sec = 1369.8, completed 10/50 warmup steps
-I0430 14:29:56.864749 140684161857344 hooks.py:117] Batch [10]:  last 10 steps exp/sec = 1369.8, completed 10/50 warmup steps
-INFO:tensorflow:loss = 9.83847 (37.378 sec)
-I0430 14:29:56.865195 140684161857344 basic_session_run_hooks.py:260] loss = 9.83847 (37.378 sec)
-INFO:tensorflow:Batch [20]:  last 10 steps exp/sec = 1364.68, completed 20/50 warmup steps
-I0430 14:30:34.378897 140147178080064 hooks.py:117] Batch [20]:  last 10 steps exp/sec = 1364.68, completed 20/50 warmup steps
-INFO:tensorflow:loss = 9.549045 (37.518 sec)
-```
diff --git a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/language_translation/tensorflow/transformer_mlperf/training/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index baab4526e..000000000
--- a/benchmarks/language_translation/tensorflow/transformer_mlperf/training/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,186 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files and checkpoints will be written>
-```
-
-Transformer Language in mlperf benchmark can run with full training or
-fewer training steps. During training we can control if it will do the evaluation
-or not.
-
-Note users can specify batch size suitable to their systems with command option `--batch-size` to achieve the best performance. If users don't specify the batch size, then the model will choose default batch size for training, which is set to 5120 in the model.
-For training with all epochs, with evaluation, and with saving checkpoints (for convergence):
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image>  \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=0 steps_between_eval=0 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50
-```
-
-For training with fewer training steps (200 steps), and with no evaluation:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image>  \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="Yes" do_eval="No" print_iter=50
-```
-
-For training with fewer training steps (200 steps), and with evaluation:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image>  \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50 \
-    bleu_source=/home/<user>/newstest2014.en --bleu_ref=/home/<user>/newstest2014.de
-```
-
-For training only for Benchmarking, to reduce training time,
-"saving checkpoints" and "doing the evaluation" can be disabled as below:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --socket-id 0 \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image>  \
-    --batch-size ${BATCH_SIZE} \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="No" do_eval="No" print_iter=50
-```
-
-All the above commands run on a single socket of the CPU. The calculation of the number of cores uses [BaseModelInitializer.set_num_inter_intra_threads()](/benchmarks/common/base_model_init.py#L152).
-
-For training in multi-instance mode (2 sockets in a single node for example) in evaluation mode,
-where we are "saving checkpoints" and "doing the evaluation":
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image> \
-    --batch-size ${BATCH_SIZE} \
-    --num-intra-threads=26 --num-inter-threads=1 \
-    --mpi_num_processes=2 \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=0 steps_between_eval=0 params=big save_checkpoints="Yes" do_eval="Yes" print_iter=50
-```
-
-For training only in multi-instance mode (4 sockets in a single node for example) for benchmarking,
-"saving checkpoints" and "doing the evaluation" can be disabled as below:
-```
-python launch_benchmark.py \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode training \
-    --model-name transformer_mlperf \
-    --data-location ${DATASET_DIR} \
-    --docker-image <docker image>  \
-    --batch-size ${BATCH_SIZE} \
-    --num-intra-threads=26 --num-inter-threads=1 \
-    --mpi_num_processes=2 \
-    --output-dir ${OUTPUT_DIR} \
-    -- random_seed=11 train_steps=200 steps_between_eval=200 params=big save_checkpoints="No" do_eval="No" print_iter=50
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output. The above two command assumes 2 sockets in the CPU,
-and 28 cores per socket. We keep aside 2 cores for horovod communications. Thus,
-in general the num-intra-threads is equal to "available cores per socket - 2".
-The transformer model related parameters is appended after "-- "
-
-The log file is saved to the `$OUTPUT_DIR` directory. Below are
-examples of what the tail of your log file should look like for the
-different configs.
-
-Single Instance sample log:
-Example log tail when running training without saving checkpoints and not doing the evaluations:
-```
-I0820 18:29:53.357862 140273402423104 basic_session_run_hooks.py:260] loss = 7.9346876 (190.364 sec)
-INFO:tensorflow:Loss for final step: 7.4975414.
-I0820 18:33:01.786211 140273402423104 estimator.py:350] Loss for final step: 7.4975414.
-:::MLPv0.5.0 transformer 1597948381.787289143 (transformer/transformer_main.py:351) eval_start
-I0820 18:33:01.787313 140273402423104 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597948381.787289143 (transformer/transformer_main.py:351) eval_start
-:::MLPv0.5.0 transformer 1597948381.787651062 (transformer/transformer_main.py:367) eval_stop
-I0820 18:33:01.787661 140273402423104 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597948381.787651062 (transformer/transformer_main.py:367) eval_stop
-:::MLPv0.5.0 transformer 1597948381.787932158 (transformer/transformer_main.py:451) run_stop
-I0820 18:33:01.787940 140273402423104 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597948381.787932158 (transformer/transformer_main.py:451) run_stop
-:::MLPv0.5.0 transformer 1597948381.788205624 (transformer/transformer_main.py:452) run_final
-I0820 18:33:01.788213 140273402423104 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597948381.788205624 (transformer/transformer_main.py:452) run_final
-num_inter_threads: 1
-num_intra_threads: 28
-Received these standard args: Namespace(accuracy_only=False, backbone_model=None, batch_size=-1, benchmark_dir='/workspace/benchmarks', benchmark_only=True, bleu_ref=None, bleu_source=None, bleu_threshold=None, checkpoint=None, data_location='/dataset', data_num_inter_threads=None, data_num_intra_threads=None, disable_tcmalloc=True, do_eval='No', epochs_between_eval=1, experimental_gelu=False, framework='tensorflow', input_graph=None, intelai_models='/workspace/intelai_models', learning_rate=2, mode='training', model_args=[], model_name='transformer_mlperf', model_source_dir='/workspace/models', mpi=None, mpi_hostnames=None, num_cores=-1, num_cpu_cores=4, num_instances=1, num_inter_threads=1, num_intra_threads=28, num_mpi=1, num_train_steps=1, optimized_softmax=True, output_dir='/workspace/benchmarks/common/tensorflow/logs', output_results=False, params='big', precision='fp32', print_iter=50, random_seed=11, save_checkpoints='No', save_profile='', socket_id=0, static_batch='No', steps_between_eval=200, tcmalloc_large_alloc_report_threshold=2147483648, tf_serving_version='master', train_epochs=None, train_steps=200, use_case='language_translation', verbose=True)
-Received these custom args: ['--random_seed=11', '--params=big', '--train_steps=200', '--steps_between_eval=200', '--do_eval=No', '--save_checkpoints=No', '--print_iter=50', '--save_profile=']
-Current directory: /workspace/benchmarks
-Running: numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/training/fp32/transformer/transformer_main.py --data_dir=/dataset --random_seed=11 --params=big --train_steps=200 --steps_between_eval=200 --do_eval=No --save_checkpoints=No --save_profile= --print_iter=50 --inter_op_parallelism_threads=1 --intra_op_parallelism_threads=28 --learning_rate=2 --static_batch=No
-PYTHONPATH: :/workspace/intelai_models_common:/workspace/intelai_models:/workspace/benchmarks/common/tensorflow:/workspace/benchmarks
-RUNCMD: python common/tensorflow/run_tf_benchmark.py --framework=tensorflow --use-case=language_translation --model-name=transformer_mlperf --precision=fp32 --mode=training --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --num-train-steps=1  --benchmark-only  --verbose --model-source-dir=/workspace/models --data-location=/dataset --disable-tcmalloc=True --random_seed=11 --params=big --train_steps=200 --steps_between_eval=200 --do_eval=No --save_checkpoints=No
-    --print_iter=50 --save_profile=
-Log file location: /tmp/models/benchmarks/common/tensorflow/logs/benchmark_transformer_mlperf_training_fp32_20200820_181929.log
-```
-
-Multi-instance sample log:
-Running 2 instances only. Both instances printing loss values.
-
-```
-I0820 20:15:17.018405 140439206057792 basic_session_run_hooks.py:260] loss = 7.363133 (230.673 sec)
-INFO:tensorflow:Loss for final step: 7.381052.
-I0820 20:19:05.257017 140439206057792 estimator.py:350] Loss for final step: 7.381052.
-:::MLPv0.5.0 transformer 1597954745.258936405 (transformer/transformer_main.py:351) eval_start
-I0820 20:19:05.258977 140439206057792 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.258936405 (transformer/transformer_main.py:351) eval_start
-:::MLPv0.5.0 transformer 1597954745.259335518 (transformer/transformer_main.py:367) eval_stop
-I0820 20:19:05.259346 140439206057792 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.259335518 (transformer/transformer_main.py:367) eval_stop
-:::MLPv0.5.0 transformer 1597954745.259663582 (transformer/transformer_main.py:451) run_stop
-I0820 20:19:05.259672 140439206057792 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.259663582 (transformer/transformer_main.py:451) run_stop
-:::MLPv0.5.0 transformer 1597954745.259935141 (transformer/transformer_main.py:452) run_final
-I0820 20:19:05.259943 140439206057792 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.259935141 (transformer/transformer_main.py:452) run_final
-INFO:tensorflow:Loss for final step: 7.128895.
-I0820 20:19:05.259997 140209294473024 estimator.py:350] Loss for final step: 7.128895.
-:::MLPv0.5.0 transformer 1597954745.261870861 (transformer/transformer_main.py:351) eval_start
-I0820 20:19:05.261909 140209294473024 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.261870861 (transformer/transformer_main.py:351) eval_start
-:::MLPv0.5.0 transformer 1597954745.262327194 (transformer/transformer_main.py:367) eval_stop
-I0820 20:19:05.262341 140209294473024 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.262327194 (transformer/transformer_main.py:367) eval_stop
-:::MLPv0.5.0 transformer 1597954745.262665272 (transformer/transformer_main.py:451) run_stop
-I0820 20:19:05.262675 140209294473024 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.262665272 (transformer/transformer_main.py:451) run_stop
-:::MLPv0.5.0 transformer 1597954745.262959242 (transformer/transformer_main.py:452) run_final
-I0820 20:19:05.262968 140209294473024 mlperf_log.py:134] :::MLPv0.5.0 transformer 1597954745.262959242 (transformer/transformer_main.py:452) run_final
-num_inter_threads: 1
-num_intra_threads: 28
-Received these standard args: Namespace(accuracy_only=False, backbone_model=None, batch_size=-1, benchmark_dir='/workspace/benchmarks', benchmark_only=True, bleu_ref=None, bleu_source=None, bleu_threshold=None, checkpoint=None, data_location='/dataset', data_num_inter_threads=None, data_num_intra_threads=None, disable_tcmalloc=True, do_eval='No', epochs_between_eval=1, experimental_gelu=False, framework='tensorflow', input_graph=None, intelai_models='/workspace/intelai_models', learning_rate=2, mode='training', model_args=[], model_name='transformer_mlperf', model_source_dir='/workspace/models', mpi=None, mpi_hostnames=None, num_cores=-1, num_cpu_cores=4, num_instances=1, num_inter_threads=1, num_intra_threads=28, num_mpi=1, num_train_steps=1, optimized_softmax=True, output_dir='/workspace/benchmarks/common/tensorflow/logs', output_results=False, params='big', precision='fp32', print_iter=50, random_seed=11, save_checkpoints='No', save_profile='', socket_id=0, static_batch='No', steps_between_eval=200, tcmalloc_large_alloc_report_threshold=2147483648, tf_serving_version='master', train_epochs=None, train_steps=200, use_case='language_translation', verbose=True)
-Received these custom args: ['--random_seed=11', '--params=big', '--train_steps=200', '--steps_between_eval=200', '--do_eval=No', '--save_checkpoints=No', '--print_iter=50', '--save_profile=']
-Current directory: /workspace/benchmarks
-Running: numactl --cpunodebind=0 --membind=0 mpirun --allow-run-as-root -n 2 --map-by socket python /workspace/intelai_models/training/fp32/transformer/transformer_main.py --data_dir=/dataset --random_seed=11 --params=big --train_steps=200 --steps_between_eval=200 --do_eval=No --save_checkpoints=No --save_profile= --print_iter=50 --inter_op_parallelism_threads=1 --intra_op_parallelism_threads=28 --learning_rate=2 --static_batch=No
-PYTHONPATH: :/workspace/intelai_models_common:/workspace/intelai_models:/workspace/benchmarks/common/tensorflow:/workspace/benchmarks
-RUNCMD: python common/tensorflow/run_tf_benchmark.py --framework=tensorflow --use-case=language_translation --model-name=transformer_mlperf --precision=fp32 --mode=training --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --num-train-steps=1  --benchmark-only  --verbose --model-source-dir=/workspace/models --data-location=/dataset --disable-tcmalloc=True --random_seed=11 --params=big --train_steps=200 --steps_between_eval=200 --do_eval=No --save_checkpoints=No
-    --print_iter=50 --save_profile=
-Log file location: /tmp/models/benchmarks/common/tensorflow/logs/benchmark_transformer_mlperf_training_fp32_20200820_200237.log
-```
diff --git a/benchmarks/launch_benchmark.py b/benchmarks/launch_benchmark.py
index 405d453a8..89a1da3a5 100644
--- a/benchmarks/launch_benchmark.py
+++ b/benchmarks/launch_benchmark.py
@@ -140,6 +140,15 @@ def validate_args(self):
         if self.args.mode == "inference" and self.args.checkpoint:
             print("Warning: The --checkpoint argument is being deprecated in favor of using frozen graphs.")
 
+        # if itex is installed, OneDNN Graph is enabled by default.
+        try:
+            import intel_extension_for_tensorflow as itex
+            print("Intel Extension for Tensorflow " + itex.__version__ + " is installed.")
+            if self.args.onednn_graph is None and self.args.precision == "int8":
+                self.args.onednn_graph = True
+        except ModuleNotFoundError:
+            self.args.onednn_graph = False
+
     def get_model_use_case(self, benchmark_scripts, os_type):
         """
         Infers the use case based on the directory structure for the specified model.
@@ -242,7 +251,8 @@ def get_env_vars(self, benchmark_scripts, use_case, intelai_models,
             "VERBOSE": args.verbose,
             "WEIGHT_SHARING": args.weight_sharing,
             "SYNTHETIC_DATA": args.synthetic_data,
-            "GPU": str(args.gpu)
+            "GPU": str(args.gpu),
+            "ONEDNN_GRAPH": str(args.onednn_graph)
         }
 
         # Add custom model args as env vars)
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 510aa6adb..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,113 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model checkpoints, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export PRETRAINED_MODEL=<path to the pretrained model checkpoints directory>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-> If you are going to run on bare metal (without the `--docker-image` flag in the
-> commands below) you will need to update a couple of paths in the
-> `$PRETRAINED_MODEL/pipeline.config` file:
-> ```
-> # Replace in the pretrained model path on line 128
-> line_128=$(sed -n '128p' ${PRETRAINED_MODEL}/pipeline.config)
-> new_line_128="  label_map_path: \"$PRETRAINED_MODEL/mscoco_label_map.pbtxt\""
-> sed -i.bak "128s+$line_128+$new_line_128+" ${PRETRAINED_MODEL}/pipeline.config
->
-> # Replace in the dataset directory on line 132
-> line_132=$(sed -n '132p' ${PRETRAINED_MODEL}/pipeline.config)
-> new_line_132="    input_path: \"$DATASET_DIR/coco_val.record\""
-> sed -i.bak "132s+$line_132+$new_line_132+" ${PRETRAINED_MODEL}/pipeline.config
-> ```
-> If you run in a container using the `--docker-image` flag do not edit this
-> file, since the dataset and checkpoints will be mounted to the location
-> that the original `pipeline.config` expects.
-
-Run batch and online inference using the following command:
-```
-python launch_benchmark.py \
- --data-location ${DATASET_DIR} \
- --model-source-dir ${TF_MODELS_DIR} \
- --model-name faster_rcnn \
- --framework tensorflow \
- --precision fp32 \
- --mode inference \
- --socket-id 0 \
- --checkpoint ${PRETRAINED_MODEL} \
- --docker-image <docker image> \
- --output-dir ${OUTPUT_DIR} \
- -- config_file=pipeline.config
-```
-
-Test accuracy using the following:
-```
-python launch_benchmark.py \
- --model-name faster_rcnn \
- --mode inference \
- --precision fp32 \
- --framework tensorflow \
- --docker-image <docker image> \
- --model-source-dir ${TF_MODELS_DIR} \
- --data-location ${DATASET_DIR} \
- --in-graph ${PRETRAINED_MODEL}/frozen_inference_graph.pb \
- --output-dir ${OUTPUT_DIR} \
- --accuracy-only
-```
-
-Output files and logs are saved to the `${OUTPUT_DIR}` directory.
-
-Below is a sample log file tail when running for batch and online inference:
-```
-I0923 18:13:53.033420 140562203281216 eval_util.py:72] Metrics written to tf summary.
-I0923 18:13:53.033456 140562203281216 eval_util.py:463] Finished evaluation!
-Time spent : ... seconds.
-Time spent per BATCH: ... seconds.
-Log file location: {--output-dir value}/benchmark_faster_rcnn_inference_fp32_20200923_181013.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.35s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.489
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.355
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_fp32_20190114_205714.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.35s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.489
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.355
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_fp32_20190114_205714.log
-```
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md
index 4d574b273..a27e030a7 100644
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md
+++ b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md
@@ -24,23 +24,23 @@ and preprocessing the COCO validation dataset.
 | [fp32_inference.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh) | Runs batch and online inference using the coco dataset |
 | [fp32_accuracy.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh) | Runs inference and evaluates the model's accuracy |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.6 or 3.7
         <li>git
@@ -55,7 +55,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>pillow>=9.3.0
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -98,7 +98,7 @@ above, set environment variables for the path to your `DATASET_DIR` (directory
 where the `coco_val.record` TF records file is located)  and an `OUTPUT_DIR`
 where log files will be written, then run a [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the directory that contains the coco_val.record file>
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index d16d03071..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,80 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-Run batch and online inference using the following command.
-```
-python launch_benchmark.py \
- --data-location ${DATASET_DIR} \
- --model-source-dir ${TF_MODELS_DIR} \
- --model-name faster_rcnn \
- --framework tensorflow \
- --precision int8 \
- --mode inference \
- --socket-id 0 \
- --in-graph ${PRETRAINED_MODEL} \
- --docker-image <docker image> \
- --output-dir ${OUTPUT_DIR} \
- --benchmark-only \
- -- number_of_steps=5000
-```
-
-Test accuracy where the `--data-location` is the path the directory
-where your `coco_val.record` file is located and the `--in-graph` is
-the pre-trained graph model:
-```
-python launch_benchmark.py \
- --model-name faster_rcnn \
- --mode inference \
- --precision int8 \
- --framework tensorflow \
- --socket-id 0 \
- --docker-image <docker image> \
- --model-source-dir ${TF_MODELS_DIR} \
- --data-location ${DATASET_DIR}/coco_val.record \
- --in-graph ${PRETRAINED_MODEL} \
- --output-dir ${OUTPUT_DIR} \
- --accuracy-only
-```
-
-Output files and logs are saved to the `${OUTPUT_DIR}` directory.
-
-Below is a sample log file tail when running for batch and online inference:
-```
-Step 4970: ... seconds
-Step 4980: ... seconds
-Step 4990: ... seconds
-Avg. Duration per Step: ...
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_int8_20190117_232539.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-Accumulating evaluation results...
-DONE (t=1.34s).
-Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.310
-Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.479
-Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.351
-Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.310
-Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
-Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.267
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.372
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.375
-Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.375
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
-Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_int8_20190117_231937.log
-```
\ No newline at end of file
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md
index e5b5425d0..226973f67 100644
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md
+++ b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md
@@ -31,23 +31,23 @@ These quickstart scripts can be run in different environments:
 * [Docker](#docker)
 * [Kubernetes](#kubernetes)
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.6 or 3.7
         <li>git
@@ -62,7 +62,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>pillow>=9.3.0
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -103,7 +103,7 @@ above, set environment variables for the path to your `DATASET_DIR` (where
 the coco TF records file is located) and an `OUTPUT_DIR` where log files
 will be written, then run a [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the coco dataset>
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/README.md b/benchmarks/object_detection/tensorflow/rfcn/inference/README.md
index 72908a31f..daf51df3b 100644
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/README.md
+++ b/benchmarks/object_detection/tensorflow/rfcn/inference/README.md
@@ -26,21 +26,21 @@ downloading and preprocessing the COCO validation dataset.
 | [`inference.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
 | [`accuracy.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh) | Processes the TF records to run inference and check accuracy on the results. |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>git
         <li>numactl
@@ -59,7 +59,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li>git
@@ -75,14 +75,14 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pillow>=9.3.0
         <li>protobuf-compiler
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
         <li>build-essential
         <li>Cython
         <li>contextlib2
@@ -92,7 +92,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pillow>=9.3.0
         <li>protobuf-compiler
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -117,7 +117,7 @@ export PRETRAINED_MODEL=$(pwd)/rfcn_final_fused_pad_and_conv.pb
 ```
 
 RFCN uses the object detection code from the [TensorFlow Model Garden](https://github.com/tensorflow/models).
-Clone this repo with the SHA specified below and apply the patch from the model zoo directory.
+Clone this repo with the SHA specified below and apply the patch from the AI Reference Models directory.
 Set the `TF_MODELS_DIR` environment variable to the path of your clone of the TF Model Garden.
 ```
 # Clone the TensorFlow Model Garden
@@ -125,8 +125,8 @@ git clone https://github.com/tensorflow/models.git tensorflow-models
 cd tensorflow-models
 git checkout 6c21084503b27a9ab118e1db25f79957d5ef540b
 
-# Apply the TF2 patch from the model zoo repo directory
-git apply --ignore-space-change --ignore-whitespace <model zoo directory>/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
+# Apply the TF2 patch from the AI Reference Models repo directory
+git apply --ignore-space-change --ignore-whitespace <AI Reference Models directory>/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
 
 # Set the TF_MODELS_DIR env var
 export TF_MODELS_DIR=$(pwd)
@@ -141,13 +141,13 @@ protoc object_detection/protos/*.proto --python_out=.
 cd ../..
 ```
 
-Once your environment is setup, navigate back to your Model Zoo directory. Ensure that
+Once your environment is setup, navigate back to your AI Reference Models directory. Ensure that
 you have set environment variables pointing to the TensorFlow Model Garden repo, the dataset,
 and output directories, and then run a quickstart script.
 
 To run inference with performance metrics:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
@@ -162,7 +162,7 @@ export BATCH_SIZE=<customized batch size value>
 
 To get accuracy metrics:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
@@ -188,9 +188,9 @@ cd %TF_MODELS_DIR%\research
 After installing the prerequisites and cloning the TensorFlow models repo, and downloading the pretrained model,
 set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written,
 TF_MODELS_DIR, and `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts).
+Navigate to your AI Reference Models directory and then run a [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 97d0e9aa5..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,89 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-The command below runs batch and online inference. Note that the
-`--data-location ${DATASET_DIR}` should point to the raw COCO dataset images
-(for example: `DATASET_DIR=/home/<user>/coco_dataset/val2017`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision fp32 \
-    --framework tensorflow \
-    --docker-image <docker image> \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    -- number_of_steps=500
-```
-
-Or for accuracy testing, use the command below and set the `--data-location ${DATASET_DIR}`
-to the path to the COCO validation TF records file (for example:
-`DATASET_DIR=/home/<user>/coco_output/coco_val.record`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision fp32 \
-    --framework tensorflow \
-    --docker-image <docker image> \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    -- split="accuracy_message"
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output .
-
-Log files are located at the value of `$OUTPUT_DIR`. Below is a sample log
-file tail when running for batch and online inference:
-```
-Average time per step: ... sec
-Received these standard args: Namespace(accuracy_only=False, batch_size=1, benchmark_only=False, checkpoint='/checkpoints', data_location='/dataset', framework='tensorflow', input_graph=None, intelai_models='/workspace/intelai_models', mode='inference', model_args=[], model_name='rfcn', model_source_dir='/workspace/models', num_cores=-1, num_inter_threads=2, num_intra_threads=56, precision='fp32, socket_id=0, use_case='object_detection', verbose=True)
-Received these custom args: ['--config_file=rfcn_pipeline.config']
-Run model here.
-current directory: /workspace/models/research
-Running: numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/eval.py --inter_op 1 --intra_op 28 --omp 28 --pipeline_config_path /checkpoints/rfcn_pipeline.config --checkpoint_dir /checkpoints --eval_dir /workspace/models/research/object_detection/models/rfcn/eval  --logtostderr  --blocktime=0  --run_once=True
-PYTHONPATH: :/workspace/intelai_models:/workspace/models/research:/workspace/models/research/slim:/workspace/models
-RUNCMD: python common/tensorflow/run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --data-location=/dataset --socket-id 0 --verbose --checkpoint=/checkpoints         --config_file=rfcn_pipeline.config
-Batch Size: 1
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.19s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.532
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.389
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.282
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.396
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20181221_211905.log
-```
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index c00b286a3..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,92 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset (raw images for inference or the TF records file for accuracy)>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-The command below runs batch and online inference. Note that the
-`--data-location ${DATASET_DIR}` should point to the raw COCO dataset images
-(for example `DATASET_DIR=/home/<user>/coco_dataset/val2017`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --docker-image <docker image> \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --benchmark-only \
-    --output-dir ${OUTPUT_DIR} \
-    -- number_of_steps=500
-```
-
-Or for accuracy testing, use the command below and set the `--data-location ${DATASET_DIR}`
-to the path the TF record file (for example `DATASET_DIR=/home/<user>/coco_output/coco_val.record`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --docker-image <docker image> \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --accuracy-only \
-    --output-dir ${OUTPUT_DIR} \
-    -- split="accuracy_message"
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-Log files are located at the `${OUTPUT_DIR}` path.
-
-Below is a sample log file tail when running for batch
-and online inference:
-```
-Step 0: ... seconds
-Step 10: ... seconds
-...
-Step 460: ... seconds
-Step 470: ... seconds
-Step 480: ... seconds
-Step 490: ... seconds
-Avg. Duration per Step: ...
-...
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_int8_20190416_182445.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-...
-Accumulating evaluation results...
-DONE (t=1.91s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.327
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.506
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.365
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.327
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.376
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_int8_20190227_194752.log
-```
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/README.md b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/README.md
deleted file mode 100644
index 3606bba08..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/README.md
+++ /dev/null
@@ -1,214 +0,0 @@
-<!--- 0. Title -->
-# RFCN Int8 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running RFCN Int8 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`int8_inference.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/int8/int8_inference.sh) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`int8_accuracy.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/int8/int8_accuracy.sh) | Processes the TF records to run inference and check accuracy on the results. |
-
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-For more information on the required dependencies, see the documentation on [prerequisites](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#installation)
-in the TensorFlow models repo.
-
-Download the pretrained model and set the `PRETRAINED_MODEL`
-environment variable to point to the frozen graph file.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/rfcn_resnet101_int8_coco_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/rfcn_resnet101_int8_coco_pretrained_model.pb
-```
-
-RFCN uses the object detection code from the [TensorFlow Model Garden](https://github.com/tensorflow/models).
-Clone this repo with the SHA specified below and apply the patch from the model zoo directory.
-Set the `TF_MODELS_DIR` environment variable to the path of your clone of the TF Model Garden.
-```
-# Clone the TensorFlow Model Garden
-git clone https://github.com/tensorflow/models.git tensorflow-models
-cd tensorflow-models
-git checkout 6c21084503b27a9ab118e1db25f79957d5ef540b
-
-# Apply the TF2 patch from the model zoo repo directory
-git apply --ignore-space-change --ignore-whitespace <model zoo directory>/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
-
-# Set the TF_MODELS_DIR env var
-export TF_MODELS_DIR=$(pwd)
-```
-
-### Run on Linux
-Download and install [Google Protobuf version 3.3.0](https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip), and
-run [protobuf compilation](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#protobuf-compilation).
-```
-# Protobuf compilation from the TF models research directory
-cd research
-protoc object_detection/protos/*.proto --python_out=.
-cd ../..
-```
-
-Once your environment is setup, navigate back to your Model Zoo directory. Ensure that
-you have set environment variables pointing to the TensorFlow Model Garden repo, the dataset,
-and output directories, and then run a quickstart script.
-
-To run inference with performance metrics:
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/int8/int8_inference.sh
-```
-
-To get accuracy metrics:
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/int8/int8_accuracy.sh
-```
-
-### Run on Windows
-* Download and install [Google Protobuf version 3.4.0]((https://github.com/protocolbuffers/protobuf/releases/tag/v3.4.0)) for Windows in addition to the above listed dependencies.
-Download and extract [protoc-3.4.0-win32.zip](https://github.com/protocolbuffers/protobuf/releases/download/v3.4.0/protoc-3.4.0-win32.zip)
-Navigate to the `research` directory in `TF_MODELS_DIR` and install Google Protobuf:
-```
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-cd %TF_MODELS_DIR%\research
-“C:\<user>\protoc-3.4.0-win32\bin\protoc.exe” object_detection/protos/*.proto --python_out=.
-```
-
-After installing the prerequisites and cloning the TensorFlow models repo, and downloading the pretrained model,
-set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written,
-TF_MODELS_DIR, and `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts>
-set OUTPUT_DIR=<directory where log files will be written>
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\object_detection\tensorflow\rfcn\inference\cpu\int8\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\coco_dataset\val2017`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\coco_dataset\val2017
-> /d/user/coco_dataset/val2017
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/coco_dataset/val2017`.
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [oneContainer](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-int8-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-int8-inference-tensorflow-container.html).
-
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
index 1d68d5627..8758aa3e9 100644
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
+++ b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
@@ -25,21 +25,21 @@ Set the `DATASET_DIR` to point to the dataset directory that contains the TF rec
 | [`inference_throughput_multi_instance.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_throughput_multi_instance.sh) | A multi-instance run that uses all the cores for each socket for each instance with a batch size of 448 and synthetic data. Supported versions are (fp32, int8, bfloat16, bfloat32) |
 | [`inference_realtime_multi_instance.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_realtime_multi_instance.sh) | A multi-instance run that uses 4 cores per instance with a batch size of 1. Uses synthetic data if no `DATASET_DIR` is set. Supported versions are (fp32, int8, bfloat16, bfloat32) |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -51,12 +51,13 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>pillow>=9.3.0
         <li>pycocotools
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
         <li>Activate the `tensorflow` conda environment
         <pre>conda activate tensorflow</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li>git
@@ -71,14 +72,15 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>pillow>=9.3.0
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
         <li>build-essential
         <li>Cython
         <li>contextlib2
@@ -87,7 +89,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>matplotlib
         <li>pillow>=9.3.0
         <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -108,14 +110,18 @@ export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
 # Int8 Pretrained model
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
 export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
+
+# Int8 Pretrained model for OneDNN Graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point)
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/ssd_mb_itex_int8.pb
+export PRETRAINED_MODEL=$(pwd)/ssd_mb_itex_int8.pb
 ```
 
 After installing the prerequisites and downloading the pretrained model, set the environment variables and for the `DATASET_DIR` use COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts) on either Linux or Windows.
+Navigate to your AI Reference Models directory and then run a [quickstart script](#quick-start-scripts) on either Linux or Windows.
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export PRETRAINED_MODEL=<path to the downloaded frozen graph>
@@ -131,7 +137,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the pretrained model pb file>
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/bfloat16/README.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/bfloat16/README.md
deleted file mode 100644
index 8ac94e131..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/bfloat16/README.md
+++ /dev/null
@@ -1,116 +0,0 @@
-<!--- 0. Title -->
-# SSD-MobileNet BFloat16 inference
-
-<!-- 10. Description -->
-
-This document has instructions for running SSD-MobileNet BFloat16 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-SSD-Mobilenet quickstart scripts. The accuracy quickstart script require the dataset to be converted into the TF records format.
-See the [COCO dataset](https://github.com/IntelAI/models/tree/master/datasets/coco) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-Set the `DATASET_DIR` to point to the dataset directory that contains the TF records file `coco_val.record` when running <model name> accuracy script.
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32 or bfloat16). Waits for all instances to complete, then prints a summarized throughput value. |
-| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32 or bfloat16). Waits for all instances to complete, then prints a summarized throughput value. |
-| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32 or bfloat16). |
-
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-For more information on the dependencies, see the documentation on [prerequisites](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#installation)
-in the TensorFlow models repo.
-
-Download the pretrained model and set the `PRETRAINED_MODEL` environment
-variable to the path of the frozen graph. 
-For Linux, run:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
-export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
-```
-
-After installing the prerequisites and downloading the pretrained model, set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written, `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts and `PRECISION` to bfloat16.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts) on Linux.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-# Set environment variables
-export PRETRAINED_MODEL=<path to the downloaded frozen graph>
-export DATASET_DIR=<path to the directory where coco tf record file is stored>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRECISION=bfloat16
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script (for example, inference realtime.sh)
-./quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index ab983f286..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,177 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where
-log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export PRETRAINED_MODEL=<path to the downloaded frozen graph>
-export DATASET_DIR=<path to the coco tf record file>
-export OUTPUT_DIR=<directory where log files will be written>
-```
-
-SSD-MobileNet can be run for testing online inference or testing accuracy.
-
-* To run for online inference, use the following command:
-  ```
-  python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only
-  ```
-* To test accuracy, use the following command:
-  ```
-  python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only
-  ```
-
-The log file is saved to the value of `${OUTPUT_DIR}`.
-
-Below is a sample log file tail when running for performance:
-```
-INFO:tensorflow:Processed 5001 images... moving average latency 37 ms
-INFO:tensorflow:Finished processing records
-Latency: min = 33.8, max = 6635.9, mean= 38.4, median = 37.2
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_ssd-mobilenet_inference_fp32_20190130_225108.log
-```
-
-Below is a sample log file tail when testing accuracy:
-```
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.231
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.349
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.254
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.231
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.209
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.264
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.264
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.264
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_ssd-mobilenet_inference_fp32_20190123_225145.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with synthetic data (to use real data, you can add `--data-location ${DATASET_DIR}`).
-Your output may vary from what's seen below, depending on the number of
-cores on your system.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 448.
-  ```
-  python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 448 \
-    --numa-cores-per-instance socket \
-    --docker-image <docker image> \
-    --benchmark-only
-  ```
-  The output will show the multi-instance command being run and a list of the
-  log files (one for each instance and a combined log file):
-  ```
-  Multi-instance run:
-  OMP_NUM_THREADS=28 numactl --localalloc --physcpubind=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 28 -e 1 -b 448 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores28_instance0.log 2>&1 & \
-  OMP_NUM_THREADS=28 numactl --localalloc --physcpubind=28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 28 -e 1 -b 448 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores28_instance1.log 2>&1 & \
-  wait
-
-  The following log files were saved to the output directory:
-  ssd-mobilenet_fp32_inference_bs448_cores28_instance0.log
-  ssd-mobilenet_fp32_inference_bs448_cores28_instance1.log
-
-  A combined log file was saved to the output directory:
-  ssd-mobilenet_fp32_inference_bs1_cores28_all_instances.log
-  ```
-  The following grep command can be used to get a summary of the total samples
-  per second from the combined log file:
-  ```
-  grep 'Total samples/sec' ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs448_cores*_all_instances.log  | awk -F' ' '{sum+=$3;} END{print sum} '
-  ```
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --docker-image <docker image> \
-    --benchmark-only
-  ```
-  The output will show the multi-instance command being run and a list of the
-  log files (one for each instance and a combined log file):
-  ```
-  Multi-instance run:
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=0,1,2,3 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance0.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=4,5,6,7 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance1.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=8,9,10,11 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance2.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=12,13,14,15 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance3.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=16,17,18,19 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance4.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=20,21,22,23 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance5.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=24,25,26,27 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance6.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=28,29,30,31 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance7.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=32,33,34,35 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance8.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=36,37,38,39 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance9.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=40,41,42,43 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance10.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=44,45,46,47 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance11.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=48,49,50,51 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance12.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=52,53,54,55 python /workspace/intelai_models/inference/infer_detections.py -g /in_graph/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_instance13.log 2>&1 & \
-  wait
-
-  The following log files were saved to the output directory:
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance0.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance1.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance2.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance3.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance4.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance5.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance6.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance7.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance8.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance9.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance10.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance11.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance12.log
-  ssd-mobilenet_fp32_inference_bs1_cores4_instance13.log
-
-  A combined log file was saved to the output directory:
-  ssd-mobilenet_fp32_inference_bs1_cores4_all_instances.log
-  ```
-  The following grep command can be used to get a summary of the total samples
-  per second from the combined log file:
-  ```
-  grep 'Total samples/sec' ${OUTPUT_DIR}/ssd-mobilenet_fp32_inference_bs1_cores4_all_instances.log  | awk -F' ' '{sum+=$3;} END{print sum} '
-  ```
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 849a07dd9..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,189 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where
-log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export PRETRAINED_MODEL=<path to the pretrained model pb file>
-export DATASET_DIR=<path to the coco tf record file>
-export OUTPUT_DIR=<directory where log files will be written>
-```
-
-SSD-MobileNet can be run for testing online inference or testing accuracy.
-
-* Run online inference using the following command:
-  ```
-  python launch_benchmark.py \
-    --model-name ssd-mobilenet \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    --batch-size 1
-  ```
-* Test accuracy using the following command:
-  ```
-  python launch_benchmark.py \
-    --model-name ssd-mobilenet \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --socket-id 0 \
-    --docker-image <docker image> \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    --batch-size 1
-  ```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output. The log file is saved to the value of `${OUTPUT_DIR}`.
-
-Below is a sample log file tail when running for online inference:
-```
-Step 4970: 0.0305020809174 seconds
-Step 4980: 0.0294089317322 seconds
-Step 4990: 0.0301029682159 seconds
-Avg. Duration per Step:0.0300041775227
-Avg. Duration per Step:0.0301246762276
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_ssd-mobilenet_inference_int8_20190417_175418.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-Running per image evaluation...
-Evaluate annotation type *bbox*
-DONE (t=9.53s).
-Accumulating evaluation results...
-DONE (t=1.10s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.172
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.271
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.183
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.172
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.171
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.212
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.212
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.212
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_ssd-mobilenet_inference_int8_20181204_185432.log
-```
-
-Batch and online inference can also be run with multiple instances using
-`numactl`. The following commands have examples how to do multi-instance runs
-using the `--numa-cores-per-instance` argument. Note that these examples are
-running with synthetic data (to use real data, you can add `--data-location ${DATASET_DIR}`).
-Your output may vary from what's seen below, depending on the number of
-cores on your system.
-
-* For multi-instance batch inference, the recommended configuration uses all
-  the cores on a socket for each instance (this means that if you have 2 sockets,
-  you would be running 2 instances - one per socket) and a batch size of 448.
-  ```
-  python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision int8 \
-    --batch-size 448 \
-    --numa-cores-per-instance socket \
-    --mode inference \
-    --docker-image <docker image> \
-    --benchmark-only
-  ```
-  The output will show the multi-instance command being run and a list of the
-  log files (one for each instance and a combined log file):
-  ```
-  Multi-instance run:
-  OMP_NUM_THREADS=28 numactl --localalloc --physcpubind=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 28 -e 1 -b 448 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs448_cores28_instance0.log 2>&1 & \
-  OMP_NUM_THREADS=28 numactl --localalloc --physcpubind=28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 28 -e 1 -b 448 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs448_cores28_instance1.log 2>&1 & \
-  wait
-
-  The following log files were saved to the output directory:
-  ssd-mobilenet_int8_inference_bs448_cores28_instance0.log
-  ssd-mobilenet_int8_inference_bs448_cores28_instance1.log
-
-  A combined log file was saved to the output directory:
-  ssd-mobilenet_int8_inference_bs448_cores28_all_instances.log
-
-  Ran inference with batch size 448
-  ```
-  The following grep command can be used to get a summary of the total samples
-  per second from the combined log file:
-  ```
-  grep 'Total samples/sec' ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs448_cores*_all_instances.log  | awk -F' ' '{sum+=$3;} END{print sum} '
-  ```
-* For multi-instance online inference, the recommended configuration is using
-  4 cores per instance and a batch size of 1.
-  ```
-  python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --model-name ssd-mobilenet \
-    --framework tensorflow \
-    --precision int8 \
-    --batch-size 1 \
-    --numa-cores-per-instance 4 \
-    --mode inference \
-    --docker-image <docker image> \
-    --benchmark-only
-  ```
-  The output will show the multi-instance command being run and a list of the
-  log files (one for each instance and a combined log file):
-  ```
-  Multi-instance run:
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=0,1,2,3 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance0.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=4,5,6,7 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance1.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=8,9,10,11 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance2.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=12,13,14,15 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance3.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=16,17,18,19 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance4.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=20,21,22,23 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance5.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=24,25,26,27 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance6.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=28,29,30,31 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance7.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=32,33,34,35 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance8.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=36,37,38,39 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance9.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=40,41,42,43 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance10.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=44,45,46,47 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance11.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=48,49,50,51 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance12.log 2>&1 & \
-  OMP_NUM_THREADS=4 numactl --localalloc --physcpubind=52,53,54,55 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb -i 1000 -w 200 -a 4 -e 1 -b 1 >> ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_instance13.log 2>&1 & \
-  wait
-
-  The following log files were saved to the output directory:
-  ssd-mobilenet_int8_inference_bs1_cores4_instance0.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance1.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance2.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance3.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance4.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance5.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance6.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance7.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance8.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance9.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance10.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance11.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance12.log
-  ssd-mobilenet_int8_inference_bs1_cores4_instance13.log
-
-  A combined log file was saved to the output directory:
-  ssd-mobilenet_int8_inference_bs1_cores4_all_instances.log
-  ```
-  The following grep command can be used to get a summary of the total samples
-  per second from the combined log file:
-  ```
-  grep 'Total samples/sec' ${OUTPUT_DIR}/ssd-mobilenet_int8_inference_bs1_cores4_all_instances.log  | awk -F' ' '{sum+=$3;} END{print sum} '
-  ```
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/README.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/README.md
deleted file mode 100644
index f593752f0..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/README.md
+++ /dev/null
@@ -1,156 +0,0 @@
-<!--- 0. Title -->
-# SSD-MobileNet Int8 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running SSD-MobileNet Int8 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-SSD-Mobilenet quickstart scripts. The inference and accuracy quickstart scripts require the dataset to be converted into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`int8_inference.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/int8/int8_inference.sh) | Runs inference on TF records and outputs performance metrics. |
-| [`int8_accuracy.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/int8/int8_accuracy.sh) | Runs inference and checks accuracy on the results. |
-| [`multi_instance_batch_inference.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/int8/multi_instance_batch_inference.sh) | A multi-instance run that uses all the cores for each socket for each instance with a batch size of 448 and synthetic data. |
-| [`multi_instance_online_inference.sh`](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/int8/multi_instance_online_inference.sh) | A multi-instance run that uses 4 cores per instance with a batch size of 1 and synthetic data. |
-
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-For more information see the documentation on [prerequisites](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#installation)
-in the TensorFlow models repo.
-
-Download the pretrained model and set the `PRETRAINED_MODEL` environment
-variable to the path of the frozen graph. If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
-export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
-```
-
-After installing the prerequisites and downloading the pretrained model, set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written,
-and `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts) on either Linux or Windows.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-export PRETRAINED_MODEL=<path to the pretrained model pb file>
-export DATASET_DIR=<path to the coco tf record file>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/int8/<script name>.sh
-```
-
-### Run on Windows
-Using `cmd.exe`, run:
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<path to the pretrained model pb file>
-set DATASET_DIR=<path to the coco tf record file>
-set OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\object_detection\tensorflow\ssd-mobilenet\inference\cpu\int8\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\coco_dataset\coco_val.record`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\coco_dataset\coco_val.record
-> /d/user/coco_dataset/coco_val.record
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/coco_dataset/coco_val.record`.
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [oneContainer](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-mobilenet-int8-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/ssd-mobilenet-int8-inference-tensorflow-container.html).
-
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
index a6fa55efd..9bac17353 100644
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
+++ b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
@@ -69,6 +69,9 @@ def __init__(self, args, custom_args=[], platform_util=None):
         if self.args.data_location:
             self.command_prefix += " -d {0}".format(self.args.data_location)
 
+        if self.args.onednn_graph:
+            self.command_prefix += " --onednn-graph"
+
         if self.args.accuracy_only:
             self.command_prefix += " -r"
             assert self.args.data_location, "accuracy must provide the data."
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/README.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/README.md
index 6364c9150..1564f23bc 100644
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/README.md
+++ b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/README.md
@@ -36,21 +36,21 @@ test uses synthetic dataset.
 | [multi_instance_online_inference_1200.sh](/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/multi_instance_online_inference_1200.sh) | Runs multi instance realtime inference (batch-size=1) using 4 cores per instance for the specified precision (fp32, int8 or bfloat16). Uses synthetic data with an input size of 1200x1200. Waits for all instances to complete, then prints a summarized throughput value. |
 | [multi_instance_batch_inference_1200.sh](/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/fp32/multi_instance_batch_inference_1200.sh) | Runs multi instance batch inference (batch-size=16) using 1 instance per socket for the specified precision (fp32, int8 or bfloat16). Uses synthetic data with an input size of 1200x1200. Waits for all instances to complete, then prints a summarized throughput value. |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>build-essential
         <li>git
@@ -69,12 +69,13 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pillow>=9.3.0
         <li>pycocotools
         <li>tensorflow-addons==0.18.0
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
         <li>Activate the tensorflow 2.5.0 conda environment
         <pre>conda activate tensorflow</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li>build-essential
@@ -94,14 +95,15 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pillow>=9.3.0
         <li>pycocotools
         <li>tensorflow-addons==0.18.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>intel-extension-for-tensorflow (only required when using onednn graph optimization)
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
         <li>build-essential
         <li>Cython
         <li>contextlib2
@@ -112,7 +114,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>pillow>=9.3.0
         <li>pycocotools
         <li>tensorflow-addons==0.18.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -158,6 +160,10 @@ export PRETRAINED_MODEL=$(pwd)/ssd_resnet34_int8_bs1_pretrained_model.pb
 # SSD-ResNet34 Int8 1200x1200 Pretrained model
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssd_resnet34_int8_1200x1200_pretrained_model.pb
 export PRETRAINED_MODEL=$(pwd)/ssd_resnet34_int8_1200x1200_pretrained_model.pb
+
+# SSD-ResNet34 Int8 1200x1200 Pretrained model for OneDnn Graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point)
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/ssd_rn34_itex_int8.pb
+export PRETRAINED_MODEL=$(pwd)/ssd_rn34_itex_int8.pb
 ```
 
 Set the environment variables and run quickstart script on either Linux or Windows systems. If the accuracy test is being run, then set the `DATASET_DIR` to point to the folder where the COCO dataset
@@ -165,7 +171,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # set environment variables
@@ -183,7 +189,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using `cmd.exe`,  run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<path to the 300x300 or 1200x1200 pretrained model pb file>
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 598a4f99c..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,92 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-coco validation dataset, TensorFlow models repo, pretrained model frozen graph,
-and an output directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<directory with the validation-*-of-* files (for accuracy testing only)>
-export TF_MODELS_DIR=<path to the TensorFlow Models repo>
-export PRETRAINED_MODEL=<path to the 300x300 or 1200x1200 pretrained model pb file>
-export OUTPUT_DIR=<directory where log files will be written>
-```
-
-SSD-ResNet34 can be run for testing batch or online inference, or testing accuracy.
-
-To run for batch and online inference, use the command below. If you are
-running with docker, you will also need to provide the `ssd-resnet-benchmarks`
-path for `volume` flag. To run without docker, omit the `--docker-image` and
-`--volume` flags. By default it runs with input size 300x300, you may
-add `-- input-size=1200` flag to run benchmark with input size 1200x1200.
-Use the 300x300 or 1200x1200 pretrained model, depending on the input size.
-Optionally, you can also specify the number of `warmup-steps` and `steps` as
-shown in the example below, the default values are `warmup-steps=200` and `steps=800`.
-```
-# benchmarks with input size 300x300
-python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    -- warmup-steps=50 steps=200
-```
-
-To run the accuracy test, use the command below. By default it runs with
-input size 300x300, you may add `-- input-size=1200` flag to run the test with
-input size 1200x1200. Use the 300x300 or 1200x1200 pretrained model,
-depending on the input size. To run without docker, omit the `--docker-image` and
-`--volume` flags.
-```
-# accuracy test with input size 1200x1200
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    -- input-size=1200
-```
-
-The log file is saved to the `${OUTPUT_DIR}`.
-
-Below is a sample log file tail when testing performance:
-```
-Batchsize: 1
-Time spent per BATCH:    ... ms
-Total samples/sec:    ... samples/s
-```
-
-Below is a sample log file tail when testing accuracy:
-```
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.224
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.410
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.220
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.140
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.297
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.257
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.214
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.342
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.367
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.185
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.454
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.443
-```
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 8daf9e39a..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,92 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-coco validation dataset, TensorFlow models repo, pretrained model frozen graph,
-and an output directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<directory with the validation-*-of-* files (for accuracy testing only)>
-export TF_MODELS_DIR=<path to the TensorFlow Models repo>
-export PRETRAINED_MODEL=<path to the 300x300 or 1200x1200 pretrained model pb file>
-export OUTPUT_DIR=<directory where log files will be written>
-```
-
-SSD-ResNet34 can be run for batch and online inference, or accuracy.
-
-To run for batch and online inference, use the command below.
-If you are running using docker, you will need to provide the `ssd-resnet-benchmarks`
-path for `volume` flag. To run without docker, omit the `--docker-image` and
-`--volume` flags. By default it runs with input size 300x300, you
-may add `-- input-size=1200` flag to run benchmark with input size 1200x1200.
-Use the 300x300 or 1200x1200 pretrained model, depending on the input size.
-Optionally, you can also specify the number of `warmup-steps` and `steps` as
-shown in the example below, the default values are `warmup-steps=200` and `steps=800`.
-```
-# benchmarks with input size 1200x1200
-python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    -- input-size=1200 warmup-steps=50 steps=200
-```
-
-To run the accuracy test, use the command below.
-By default it runs with  input size 300x300, you may add `-- input-size=1200`
-flag last to run the test with input size 1200x1200. Use the 300x300 or
-1200x1200 pretrained model, depending on the input size. To run without docker,
-omit the `--docker-image` and `--volume` flags.
-```
-# accuracy test with input size 300x300
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only
-```
-
-The log file is saved to the value of `${OUTPUT_DIR}`.
-
-Below is a sample log file tail when running for performance:
-```
-Batchsize: 1
-Time spent per BATCH:    ... ms
-Total samples/sec:    ... samples/s
-```
-
-Below is a sample log file tail when testing accuracy:
-```
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.211
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.366
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.216
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.220
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.344
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.216
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.302
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.310
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.091
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.334
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.494
-Current AP: 0.21082
-```
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 1ade2fbe6..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,93 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-coco validation dataset, TensorFlow models repo, pretrained model frozen graph,
-and an output directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<directory with the validation-*-of-* files (for accuracy testing only)>
-export TF_MODELS_DIR=<path to the TensorFlow Models repo>
-export PRETRAINED_MODEL=<path to the 300x300 or 1200x1200 pretrained model pb file>
-export OUTPUT_DIR=<directory where log files will be written>
-```
-
-SSD-ResNet34 can be run for testing batch or online inference, or testing accuracy.
-
-To run for batch and online inference, use the command below. If you are
-running with docker, you will also need to provide the `ssd-resnet-benchmarks`
-path for `volume` flag. To run without docker, omit the `--docker-image` and
-`--volume` flags. By default it runs with input size 300x300, you may
-add `-- input-size=1200` flag to run benchmark with input size 1200x1200.
-Use the 300x300 or 1200x1200 pretrained model, depending on the input size.
-Optionally, you can also specify the number of `warmup-steps` and `steps` as
-shown in the example below, the default values are `warmup-steps=200` and `steps=800`.
-```
-# benchmarks with input size 300x300
-python launch_benchmark.py \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision int8 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    -- warmup-steps=50 steps=200
-```
-
-To run the accuracy test, use the command below. By default it runs with
-input size 300x300, you may add `-- input-size=1200` flag to run the test with
-input size 1200x1200. Use the 300x300 or 1200x1200 pretrained model,
-depending on the input size. To run without docker, omit the `--docker-image` and
-`--volume` flags.
-```
-# accuracy test with input size 1200x1200
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision int8 \
-    --mode inference \
-    --socket-id 0 \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --volume /home/<user>/ssd-resnet-benchmarks:/workspace/ssd-resnet-benchmarks \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    -- input-size=1200
-```
-
-The log file is saved to the `${OUTPUT_DIR}`.
-
-Below is a sample log file tail when testing performance:
-```
-Batchsize: 1
-Time spent per BATCH:    ... ms
-Total samples/sec:    ... samples/s
-```
-
-Below is a sample log file tail when testing accuracy:
-```
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.204
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.360
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.208
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.213
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.335
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.210
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.294
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.301
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.083
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.327
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.484
-Current AP: 0.20408
-```
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/model_init.py b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/model_init.py
index a73caddd2..a7e1b2460 100644
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/model_init.py
+++ b/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/int8/model_init.py
@@ -84,6 +84,9 @@ def __init__(self, args, custom_args, platform_util):
         self.run_cmd += " --warmup-steps {0}".format(self.additional_args.warmup_steps)
         self.run_cmd += " --steps {0}".format(self.additional_args.steps)
 
+        if self.args.onednn_graph:
+            self.run_cmd += " --onednn-graph"
+
         if self.args.accuracy_only:
             self.run_cmd += " --accuracy-only "
             self.run_cmd += " --data-location {0}".format(self.args.data_location)
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index a425ece35..000000000
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,100 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-coco training dataset (or validation dataset for evaluation mode),
-TensorFlow models repo, and an output directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export TF_MODELS_DIR=<path to the clone of the TensorFlow models repo>
-export DATASET_DIR=<path to the COCO dataset directory>
-export OUTPUT_DIR=<directory where the log file will be written>
-```
-
-To do a demo run to test performance by training the model for a limited
-number of steps, use the command below. The `DATASET_DIR` should be pointing to
-the COCO training dataset directory. You can change the number of training
-steps or number of MPI processes to run multiple instances. If you want
-checkpoint files to be saved, specify the `--checkpoint <directory path>`
-flag for the location where files will be written.
-> Note: for best performance, use the same value for the arguments num-cores and num-intra-thread as follows:
->   For single instance run (mpi_num_processes=1): the value is equal to number of logical cores per socket.
->   For multi-instance run (mpi_num_processes > 1): the value is equal to (#_of_logical_cores_per_socket - 2).
->   If the `--num-cores` or `--num-intra-threads` args are not specified, these args will be calculated based on
->   the number of logical cores on your system.
-
-```bash
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --num-train-steps 100 \
-    --num-cores 52 \
-    --num-inter-threads 1 \
-    --num-intra-threads 52 \
-    --batch-size=100 \
-    --weight_decay=1e-4 \
-    --num_warmup_batches=20 \
-    --mpi_num_processes=1 \
-    --mpi_num_processes_per_socket=1 \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image>
-```
-
-To run training and achieve convergence, download the backbone model from the links below,
-then use the command below:
-  https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/checkpoint
-  https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/model.ckpt-28152.data-00000-of-00001
-  https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/model.ckpt-28152.index
-  https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/model.ckpt-28152.meta
-
-Place the above files in one directory, and pass that location below as --backbone-model.
-The `DATASET_DIR` should be pointing to the COCO training dataset directory. To prevent
-conficts with checkpoint files generated by previous model runs, use an empty `OUTPUT_DIR`.
-```bash
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --num-cores 50 \
-    --num-inter-threads 1 \
-    --num-intra-threads 50 \
-    --batch-size=100 \
-    --mpi_num_processes=4 \
-    --mpi_num_processes_per_socket=1 \
-    --epochs=60 \
-    --checkpoint <path to output_train_directory> \
-    --backbone-model <path to resnet34_backbone_trained_model> \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image>
-```
-
-To run in eval mode (to check accuracy) if checkpoints are available. Use the below command:
-Note that `DATASET_DIR` should now points to the location of COCO validation dataset.
-```bash
-python launch_benchmark.py \
-    --data-location ${DATASET_DIR} \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --model-name ssd-resnet34 \
-    --framework tensorflow \
-    --precision bfloat16 \
-    --mode training \
-    --num-cores 52 \
-    --num-inter-threads 1 \
-    --num-intra-threads 52 \
-    --batch-size=100 \
-    --mpi_num_processes=1 \
-    --mpi_num_processes_per_socket=1 \
-    --accuracy-only \
-    --checkpoint <path to pretrained_checkpoints> \
-    --output-dir ${OUTPUT_DIR} \
-    --docker-image <docker image>
-```
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/README.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/README.md
index 7bb014019..68b6926a1 100644
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/README.md
+++ b/benchmarks/object_detection/tensorflow/ssd-resnet34/training/bfloat16/README.md
@@ -25,20 +25,20 @@ For accuracy testing, download the COCO validation dataset, using the
 | [`bfloat16_training.sh`](/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/bfloat16/bfloat16_training.sh) | Runs multi-instance training to convergence. Download the backbone model specified in the instructions below and pass that directory path in the `BACKBONE_MODEL_DIR` environment variable. |
 | [`bfloat16_training_accuracy.sh`](/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/bfloat16/bfloat16_training_accuracy.sh) | Runs the model in eval mode to check accuracy. Specify which checkpoint files to use with the `CHECKPOINT_DIR` environment variable. |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>git
         <li>numactl
@@ -63,7 +63,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li>git
@@ -85,7 +85,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>protobuf-compiler
         <li>pycocotools
         <li>tensorflow-addons==0.18.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -100,7 +100,7 @@ for object detection models at the
 Running SSD-ResNet34 training uses code from the [TensorFlow Model Garden](https://github.com/tensorflow/models).
 Clone the repo at the commit specified below, and set the `TF_MODELS_DIR`
 environment variable to point to that directory. Apply the TF2 patch from
-the model zoo to the TensorFlow models directory.
+the AI Reference Models to the TensorFlow models directory.
 ```
 # Clone the tensorflow/models repo at the specified commit.
 # Please note that required commit for this section is different from the one used for dataset preparation.
@@ -109,8 +109,8 @@ cd tf_models
 export TF_MODELS_DIR=$(pwd)
 git checkout 8110bb64ca63c48d0caee9d565e5b4274db2220a
 
-# Apply the patch from the model zoo directory to the TensorFlow Models repo
-git apply <model zoo directory>/models/object_detection/tensorflow/ssd-resnet34/training/bfloat16/tf-2.0.diff
+# Apply the patch from the AI Reference Models directory to the TensorFlow Models repo
+git apply <AI Reference Models directory>/models/object_detection/tensorflow/ssd-resnet34/training/bfloat16/tf-2.0.diff
 
 # Protobuf compilation from the TF models research directory
 cd research
@@ -125,7 +125,7 @@ and `DATASET_DIR` (path to the COCO training dataset). Use an empty output
 directory to prevent conflicts with checkpoint files from previous runs. You can optionally
 set the `TRAIN_STEPS` (defaults to 100) and `MPI_NUM_PROCESSES` (defaults to 1).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export TF_MODELS_DIR=<path to the clone of the TensorFlow models repo>
@@ -153,7 +153,7 @@ wget -P $BACKBONE_MODEL_DIR https://storage.googleapis.com/intel-optimized-tenso
 wget -P $BACKBONE_MODEL_DIR https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/model.ckpt-28152.index
 wget -P $BACKBONE_MODEL_DIR https://storage.googleapis.com/intel-optimized-tensorflow/models/ssd-backbone/model.ckpt-28152.meta
 
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export TF_MODELS_DIR=<path to the clone of the TensorFlow models repo>
@@ -172,7 +172,7 @@ the COCO validation dataset location, and the `OUTPUT_DIR` to the location
 where log files will be written. You can optionally set the `MPI_NUM_PROCESSES`
 (defaults to 1).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export TF_MODELS_DIR=<path to the clone of the TensorFlow models repo>
diff --git a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/fp32/README.md b/benchmarks/object_detection/tensorflow/ssd-resnet34/training/fp32/README.md
index 0ae60a3d8..f77759e1b 100644
--- a/benchmarks/object_detection/tensorflow/ssd-resnet34/training/fp32/README.md
+++ b/benchmarks/object_detection/tensorflow/ssd-resnet34/training/fp32/README.md
@@ -20,20 +20,20 @@ SSD-ResNet34 training uses the COCO training dataset. Use the [instructions](htt
 | [`fp32_training.sh`](/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/fp32/fp32_training.sh) | Runs 100 training steps using mpirun for the specified number of processes (defaults to MPI_NUM_PROCESSES=1).  |
 
 <!--- 50. Bare Metal -->
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>git
         <li>numactl
@@ -57,7 +57,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li>git
@@ -78,7 +78,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>protoc
         <li>pycocotools
         <li>tensorflow-addons==0.18.0
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -111,7 +111,7 @@ from previous runs. You can optionally set the `MPI_NUM_PROCESSES` environment
 variable (defaults to 1). After all the setup is complete, run the
 [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
diff --git a/benchmarks/recommendation/tensorflow/dien/inference/README.md b/benchmarks/recommendation/tensorflow/dien/inference/README.md
index 528c48b92..e900ef070 100644
--- a/benchmarks/recommendation/tensorflow/dien/inference/README.md
+++ b/benchmarks/recommendation/tensorflow/dien/inference/README.md
@@ -36,9 +36,8 @@ Set the `DATASET_DIR` to point to the directory with the dataset files when runn
 | `inference_throughput_multi_instance.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, bfloat16 or bfloat32) with a default `batch_size=65536`. Waits for all instances to complete, then prints a summarized throughput value. |
 | `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, bfloat16 or bfloat32) with a default `batch_size=128`. |
 
-<!--- 50. AI Kit -->
 ## Run the model
-* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [AI Reference Models Github repository](https://github.com/IntelAI/models).
   ```
   git clone https://github.com/IntelAI/models.git
   ```
@@ -62,7 +61,7 @@ export PRETRAINED_MODEL=$(pwd)/dien_bf16_pretrained_opt_model.pb
 
 ## Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
diff --git a/benchmarks/recommendation/tensorflow/dien/training/README.md b/benchmarks/recommendation/tensorflow/dien/training/README.md
index 18418f1b8..1e351e138 100644
--- a/benchmarks/recommendation/tensorflow/dien/training/README.md
+++ b/benchmarks/recommendation/tensorflow/dien/training/README.md
@@ -33,9 +33,8 @@ Set the `DATASET_DIR` to point to the directory with the dataset files when runn
 |-------------|-------------|
 | [`training.sh`](/quickstart/recommendation/tensorflow/dien/training/cpu/training.sh) | Runs training with a batch size of 128 for the specified precisions fp32, bfloat16 and bfloat32. |
 
-<!--- 50. AI Kit -->
 ## Run the model
-* Follow the instructions to setup your bare metal environment on Linux systems. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
+* Follow the instructions to setup your bare metal environment on Linux systems. Ensure that you have a clone of the [AI Reference Models Github repository](https://github.com/IntelAI/models).
   ```
   git clone https://github.com/IntelAI/models.git
   ```
@@ -44,7 +43,7 @@ Set the `DATASET_DIR` to point to the directory with the dataset files when runn
 
 ## Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 # Set env vars
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index a756a80e5..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,63 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-```
-
-<model name> <precision> <mode> can be run a few different modes:
-* For batch inference, `--batch-size 256`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --batch-size 256 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image>
-  ```
-* For online inference, `--batch-size 1`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --batch-size 1 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image>
-  ```
-* For Accuracy, `--batch-size 256`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --accuracy-only \
-    --batch-size 256 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image <docker image>
-  ```
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md
index fdfe91ced..6351b20e1 100644
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md
+++ b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md
@@ -26,23 +26,23 @@ Set the `DATASET_DIR` to point to this directory when running NCF.
 | [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference (batch_size=256). |
 | [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=256). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
@@ -59,7 +59,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>tar
         <li>typing
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -92,7 +92,7 @@ the `TF_MODELS_DIR` and `PRETRAINED_MODEL` paths set from the previous commands.
 Once the environment variables are all set, you can run a
 [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset>
diff --git a/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md b/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md
index 3c10dc4ef..50908a7d6 100644
--- a/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md
+++ b/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md
@@ -27,21 +27,21 @@ python ./benchmarks/recommendation/tensorflow/wide_deep/inference/fp32/data_down
 | [`inference_online.sh`](/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/inference_online.sh) | Runs wide & deep model inference online mode (batch size = 1)|
 | [`inference_batch.sh`](/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/inference_batch.sh) | Runs wide & deep model inference in batch mode (batch size = 1024)|
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
+    <th>Setup using AI Tools on Linux</th>
+    <th>Setup without AI Tools on Linux</th>
+    <th>Setup without AI Tools on Windows</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit on Linux you will need:</p>
+      <p>To run using AI Tools on Linux you will need:</p>
       <ul>
         <li>git
         <li>numactl
@@ -51,23 +51,23 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Linux you will need:</p>
+      <p>To run without AI Tools on Linux you will need:</p>
       <ul>
         <li>Python 3
         <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
         <li>git
         <li>numactl
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit on Windows you will need:</p>
+      <p>To run without AI Tools on Windows you will need:</p>
       <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
+        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
         <li>git
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -100,7 +100,7 @@ Set the environment variables and run quickstart script on either Linux or Windo
 
 ### Run on Linux
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the Wide & Deep dataset directory>
@@ -116,7 +116,7 @@ export BATCH_SIZE=<customized batch size value>
 ### Run on Windows
 Using cmd.exe, run:
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 set PRETRAINED_MODEL=<pretrained model directory>
diff --git a/benchmarks/recommendation/tensorflow/wide_deep/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/recommendation/tensorflow/wide_deep/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 8e9a7c5cb..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,53 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-```
-
-The following commonds are examples on how <model name> can be run:
-* Running the model for online inference:
-  ```
-  python launch_benchmark.py \
-    --framework tensorflow \
-    --model-source-dir $TF_MODELS_DIR \
-    --precision fp32 \
-    --mode inference \
-    --model-name wide_deep \
-    --batch-size 1 \
-    --data-location $DATASET_DIR \
-    --checkpoint $PRETRAINED_MODEL \
-    --docker-image <docker image> \
-    --output-dir $OUTPUT_DIR \
-    --verbose
-  ```
-  The three locations used (model-source-dir, data-location, checkpoint) here,
-  works better with docker if they are located in the local disk. The locations
-  should be pointed as absolute path.
-* Running the model in batch inference mode:
-  ```
-  python launch_benchmark.py \
-    --framework tensorflow \
-    --model-source-dir $TF_MODELS_DIR \
-    --precision fp32 \
-    --mode inference \
-    --model-name wide_deep \
-    --batch-size 1024 \
-    --data-location $DATASET_DIR \
-    --checkpoint $PRETRAINED_MODEL \
-    --docker-image <docker image> \
-    --output-dir $OUTPUT_DIR \
-    --verbose
-  ```
-  The three locations used (model-source-dir, data-location, checkpoint) here,
-  works better with docker if they are located in the local disk. The locations
-  should be pointed as absolute path.
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md
index 8abc052cd..9f0deb823 100644
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md
+++ b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md
@@ -26,30 +26,30 @@ export DATASET_DIR=/home/<user>/dataset/eval_preprocessed_eval.tfrecords
 | [`online_inference.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/online_inference.sh) | Runs online inference (`batch_size=1`). The `NUM_OMP_THREADS` environment variable and the hyperparameters `num-intra-threads`, `num-inter-threads` can be tuned for best performance. If `NUM_OMP_THREADS` is not set, it will default to `1`. |
 | [`accuracy.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh) | Measures the model accuracy (`batch_size=1000`). |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support wide and deep large ds</p>
+      <p>AI Tools does not currently support wide and deep large ds</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.7 or newer
         <li><a href="https://pypi.org/project/intel-tensorflow">intel-tensorflow>=2.5.0</a>
         <li>numactl
         <li>git
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -71,7 +71,7 @@ export PRETRAINED_MODEL=$(pwd)/wide_deep_int8_pretrained_model.pb
 Set the environment variables and run quickstart script. See the list of quickstart scripts for details on the different options.
 
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to TF records file>
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 37573797f..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,112 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to TF records file>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph .pb file>
-```
-
-An accuracy test can be run using the command below:
-```
-python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --batch-size 1000 \
-    --accuracy-only \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR
-```
-
-There are a few different modes that performance tests can be run:
-* Running in online inference mode for measuring latency , set `--batch-size 1`
-  ```
-  python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --num-intra-threads 1 --num-inter-threads 1  \
-    -- num_omp_threads=1
-  ```
-* Running in batch inference mode, set `--batch-size 512`
-  By default numactl is disabled. User can specify as shown in the command below.
-  Below are the commands that gives best performance on 28 cores. The
-  hyperparameters  num-intra-threads, num-inter-threads, num_omp_threads etc
-  should be tuned for best performance.
-  * Case 1: Disabling `use_parallel_batches` option. Kmp variables can also
-    be set by using the arguments shown below or through config file
-    models/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/fp32/config.json
-    ```
-    numactl --physcpubind=0-27 -m 0 python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 512 \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --num-intra-threads 28 --num-inter-threads 1  \
-    -- num_omp_threads=20 kmp_block_time=1 kmp_settings=1
-    ```
-    The tail of the log output when the script completes should look something like this:
-    ```
-    --------------------------------------------------
-    Total test records           :  2000000
-    Batch size is                :  512
-    Number of batches            :  3907
-    Classification accuracy (%)  :  77.6693
-    No of correct predictions    :  1553386
-    Inference duration (seconds) :  6.4442
-    Average Latency (ms/batch)   :  1.6931
-    Throughput is (records/sec)  :  302410.809
-    --------------------------------------------------
-    ```
-  * Case 2: Enabling `use_parallel_batches` option.
-    ```
-    numactl --physcpubind=0-27 -m 0 python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 512 \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --num-intra-threads 1 --num-inter-threads 28 \
-    -- num_omp_threads=1 use_parallel_batches=True num_parallel_batches=28 kmp_block_time=0 kmp_settings=1
-    ```
-    The tail of the log output when the script completes should look something like this:
-    ```
-    --------------------------------------------------
-    Total test records           :  2000000
-    Batch size is                :  512
-    Number of batches            :  3907
-    Classification accuracy (%)  :  77.6693
-    No of correct predictions    :  1553386
-    Inference duration (seconds) :  3.4655
-    Average Latency (ms/batch)   :  24.8406
-    Throughput is (records/sec)  :  577120.456
-    --------------------------------------------------
-    Ran inference with batch size 512
-    Log location outside container: ${OUTPUT_DIR}/benchmark_wide_deep_large_ds_inference_fp32_20190225_062206.log
-    ```
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/int8/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/int8/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 0725c481b..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/int8/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,117 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to TF records file>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph .pb file>
-```
-
-<model name> can be run in a few different modes:
-* An accuracy test can be run using the command below.
-  ```
-  python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --batch-size 1000 \
-    --accuracy-only \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR
-  ```
-* Run an online inference test to measure latency with `--batch-size 1`.
-  ```
-  python launch_benchmark.py \
-    --model-name wide_deep_large_ds \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --docker-image <docker image> \
-    --in-graph $PRETRAINED_MODEL \
-    --data-location $DATASET_DIR \
-    --output-dir $OUTPUT_DIR \
-    --num-intra-threads 1 --num-inter-threads 1  \
-    -- num_omp_threads=1
-  ```
-* Run a batch inference test by setting `--batch-size 512`. The `numactl`
-  command is a utility which can be used to control NUMA policy for processes
-  or shared memory. To install numactl do:
-  ```
-  apt install numactl
-  ```
-  By default numactl is disabled. User can use NUMA policy control only on
-  bare metal in advanced cases to specify number of cores to be used. This can
-  be specified as shown in the command below only on bare metal. Below are the
-  commands that gives best performance on 28 cores. The hyperparameters
-  num-intra-threads, num-inter-threads, num_omp_threads etc should be tuned
-  for best performance.
-  * Case 1: Disabling `use_parallel_batches` option. In this case the batches
-    are inferred in sequential order. By default `use_parallel_batches` is
-    disabled. Kmp variables can also be set by using the arguments shown
-    below or through config file
-    `models/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/int8/config.json`
-    ```
-    numactl --physcpubind=0-27 -m 0 python launch_benchmark.py \
-        --model-name wide_deep_large_ds \
-        --precision int8 \
-        --mode inference \
-        --framework tensorflow \
-        --benchmark-only \
-        --batch-size 512 \
-        --in-graph $PRETRAINED_MODEL \
-        --data-location $DATASET_DIR \
-        --output-dir $OUTPUT_DIR \
-        --num-intra-threads 28 --num-inter-threads 1 \
-        -- num_omp_threads=16  kmp_block_time=1 kmp_settings=1
-    ```
-    The tail of the log output when the script completes should look something like this:
-    ```
-    --------------------------------------------------
-    Total test records           :  2000000
-    Batch size is                :  512
-    Number of batches            :  3907
-    Classification accuracy (%)  :  77.636
-    Inference duration (seconds) :  3.1534
-    Average Latency (ms/batch)   :  0.8285
-    Throughput is (records/sec)  :  617992.762
-    --------------------------------------------------
-    ```
-  * Case 2: Enabling `use_parallel_batches` option. In this case multiples
-    batches are inferred in parallel. Number of batches to be executed in parallel
-    can be given by argument num_parallel_batches.
-    ```
-    numactl --physcpubind=0-27 -m 0 python launch_benchmark.py \
-        --model-name wide_deep_large_ds \
-        --precision int8 \
-        --mode inference \
-        --framework tensorflow \
-        --benchmark-only \
-        --batch-size 512 \
-        --in-graph $PRETRAINED_MODEL \
-        --data-location $DATASET_DIR \
-        --output-dir $OUTPUT_DIR \
-        --num-intra-threads 1 --num-inter-threads 28  \
-        -- num_omp_threads=1 use_parallel_batches=True num_parallel_batches=28 kmp_block_time=0 kmp_settings=1
-    ```
-    The tail of the log output when the script completes should look something like this:
-    ```
-    --------------------------------------------------
-    Total test records           :  2000000
-    Batch size is                :  512
-    Number of batches            :  3907
-    Inference duration (seconds) :  1.7056
-    Average Latency (ms/batch)   :  12.2259
-    Throughput is (records/sec)  :  1172597.21
-    --------------------------------------------------
-    ```
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md
index b8f65c8ef..e5495c37f 100644
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md
+++ b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md
@@ -34,20 +34,20 @@ The `DATASET_DIR` environment variable will be used as the dataset directory whe
 | [`training.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh) | Trains the model for 10 epochs. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
 | [`training_demo.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh) | A short demo run that trains the model for 100 steps. |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>To run using AI Kit you will need:</p>
+      <p>To run using AI Tools you will need:</p>
       <ul>
         <li>numactl
         <li>wget
@@ -56,14 +56,14 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
       </ul>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3
         <li>intel-tensorflow>=2.5.0
         <li>numactl
         <li>git
         <li>wget
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -73,12 +73,12 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
 After the setup is complete, set environment variables for the path to your
 dataset directory and an output directory where logs will be written. You can
 optionally provide a directory where checkpoint files will be read and
-written. Navigate to your model zoo directory, then select a
+written. Navigate to your AI Reference Models directory, then select a
 [quickstart script](#quick-start-scripts) to run. Note that some quickstart
 scripts might use other environment variables in addition to the ones below,
 like `STEPS` and `TARGET_ACCURACY` for the `fp32_training_check_accuracy.sh` script.
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export DATASET_DIR=<path to the dataset directory>
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index 7b5de5f9b..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,41 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset (where train.csv and eval.csv are located), an output directory where log
-files will be written, and optionally a directory where checkpoint files can
-be read and written from.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset directory>
-export OUTPUT_DIR=<directory where the logs and the saved model will be written>
-export CHECKPOINT_DIR=<Optional directory where checkpoint files will be read and written>
-```
-
-Train the model (The model will be trained for 10 epochs if -- steps is not specified)
-```
-python launch_benchmark.py \
-  --model-name wide_deep_large_ds \
-  --precision fp32 \
-  --mode training  \
-  --framework tensorflow \
-  --batch-size 512 \
-  --data-location ${DATASET_DIR} \
-  --checkpoint ${CHECKPOINT_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --docker-image <docker image>
-```
-Once the training completes successfully the path of checkpoint files and saved_model.pb will be printed as shown below
-```
-INFO:tensorflow:SavedModel written to: $OUTPUT_DIR/temp-1602670603/saved_model.pb
-Using TensorFlow version 2.4.0
-Begin training and evaluation
-Saving model checkpoints to $CHECKPOINT_DIR
-****Computing statistics of train dataset*****
-estimator built
-fit done
-evaluate done
-Model exported to $OUTPUT_DIR
-```
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
deleted file mode 100644
index be35de848..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/.docs/advanced/launch_benchmark_instructions.md
+++ /dev/null
@@ -1,56 +0,0 @@
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model checkpoints>
-export TF_WAVENET_DIR=<path to your clone of the TensorFlow WaveNet repo>
-```
-
-Start a model run by executing the launch script and passing args
-specifying that we are running wavenet fp32 inference using TensorFlow,
-along with a docker image that includes Intel Optimizations for TensorFlow
-and the path to your clone of the WaveNet repo and the checkpoint
-files that were downloaded.  We are also passing a couple of extra model args
-for wavenet: the name of the checkpoint to use and the sample number.
-
-```
-python launch_benchmark.py \
-    --precision fp32 \
-    --model-name wavenet \
-    --mode inference \
-    --framework tensorflow \
-    --socket-id 0 \
-    --num-cores 1 \
-    --docker-image <docker image> \
-    --model-source-dir ${TF_WAVENET_DIR} \
-    --checkpoint ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    -- checkpoint_name=model.ckpt-99 sample=8510
-```
-
-The logs are displayed in the console output as well as saved to a
-file in the value of `${OUTPUT_DIR}`.
-
-The tail of the log should look something like this:
-```
-Time per 500 Samples: ... sec
-Samples / sec: ...
-msec / sample: ...
-Sample: 8000
-Time per 500 Samples: ... sec
-Samples / sec: ...
-msec / sample: ...
-Sample: 8500
-
-Average Throughput of whole run: Samples / sec: ...
-Average Latency of whole run: msec / sample: ...
-Finished generating. The result can be viewed in TensorBoard.
-Log file location: ${OUTPUT_DIR}/benchmark_wavenet_inference_fp32_20210601_143852.log
-```
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md
index 393e81a7d..0fc54f963 100644
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md
+++ b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md
@@ -14,23 +14,23 @@ Intel-optimized TensorFlow.
 |-------------|-------------|
 | [fp32_inference.sh](/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh) | Runs inference with a pretrained model |
 
-<!--- 50. AI Kit -->
+<!--- 50. AI Tools -->
 ## Run the model
 
 Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
+using [AI Tools](/docs/general/tensorflow/AITools.md):
 
 <table>
   <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
+    <th>Setup using AI Tools</th>
+    <th>Setup without AI Tools</th>
   </tr>
   <tr>
     <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
+      <p>AI Tools does not currently support TF 1.15.2 models</p>
     </td>
     <td>
-      <p>To run without AI Kit you will need:</p>
+      <p>To run without AI Tools you will need:</p>
       <ul>
         <li>Python 3.6 or 3.7
         <li>git
@@ -38,7 +38,7 @@ using [AI Kit](/docs/general/tensorflow/AIKit.md):
         <li>wget
         <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
         <li>librosa==0.5
-        <li>A clone of the Model Zoo repo<br />
+        <li>A clone of the AI Reference Models repo<br />
         <pre>git clone https://github.com/IntelAI/models.git</pre>
       </ul>
     </td>
@@ -68,12 +68,12 @@ tar -xvf wavenet_fp32_pretrained_model.tar.gz
 export PRETRAINED_MODEL=$(pwd)/wavenet_checkpoints
 ```
 
-Navigate to the your model zoo directory, then set an environment variable
+Navigate to the your AI Reference Models directory, then set an environment variable
 for an `OUTPUT_DIR` directory where logs will be written and ensure that you
 have the `TF_WAVENET_DIR` and `PRETRAINED_MODEL` variables set. Once this
 setup is done, you can run the `fp32_inference.sh` [quickstart script](#quick-start-scripts).
 ```
-# cd to your model zoo directory
+# cd to your AI Reference Models directory
 cd models
 
 export OUTPUT_DIR=<path to the directory where log files will be written>
diff --git a/datasets/dataset_api/conda/description.md b/datasets/dataset_api/conda/description.md
new file mode 100644
index 000000000..9f9c67465
--- /dev/null
+++ b/datasets/dataset_api/conda/description.md
@@ -0,0 +1,21 @@
+Dataset librarian is a tool to download and apply the preprocessing needed for the list of supported datasets.
+
+***
+
+### Prerequisites 
+Have either python  `3.9` or `3.10` already installed.
+
+***
+
+### Installation Command 
+```bash
+conda install dataset_librarian -c intel -c conda-forge 
+```
+
+***
+
+### PyPI Package
+[Here](https://pypi.org/project/dataset-librarian/ )
+
+***
+
diff --git a/datasets/dataset_api/conda/recipe/meta.yaml b/datasets/dataset_api/conda/recipe/meta.yaml
index 7465d50bb..00bee37fd 100644
--- a/datasets/dataset_api/conda/recipe/meta.yaml
+++ b/datasets/dataset_api/conda/recipe/meta.yaml
@@ -1,5 +1,5 @@
 {% set name = "dataset_librarian" %}
-{% set version = "1.0.2" %}
+{% set version = "1.0.4" %}
 
 package:
   name: {{ name|lower }}
@@ -7,7 +7,7 @@ package:
 
 source:
   url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/dataset_librarian-{{ version }}.tar.gz
-  sha256: 06c30f838ca668d7b3eaab51eed8e0d364adb8433210016eb7c0ae993c7ef040
+  sha256: d969f7eac1459e052d6f4e1ed541e3daf5184b260081ae9edcb2f54bc7a69544
 
 build:
   noarch: python
diff --git a/datasets/dataset_api/pyproject.toml b/datasets/dataset_api/pyproject.toml
index b5f38d687..541bd15c0 100644
--- a/datasets/dataset_api/pyproject.toml
+++ b/datasets/dataset_api/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "dataset_librarian"
-version = "1.0.2"
+version = "1.0.4"
 requires-python = ">=3.9,<3.11"
 authors = [
     { name="IntelAI", email="IntelAI@intel.com"}
diff --git a/datasets/dataset_api/src/dataset_librarian/scripts/mvtec-ad/csv_generator_mvtec.py b/datasets/dataset_api/src/dataset_librarian/scripts/mvtec-ad/csv_generator_mvtec.py
index cc287f7f8..9cec8e3be 100644
--- a/datasets/dataset_api/src/dataset_librarian/scripts/mvtec-ad/csv_generator_mvtec.py
+++ b/datasets/dataset_api/src/dataset_librarian/scripts/mvtec-ad/csv_generator_mvtec.py
@@ -34,13 +34,13 @@ def main(args):
             path_category = os.path.join(base_path,category)
             path = os.path.join(path_category,dataset,'**','*')
             if dataset == 'test':
-                with open(os.path.join(path_category,category+'_all.csv'), 'w', newline='') as file:
+                with open(os.path.join(base_path,category+'_all.csv'), 'w', newline='') as file:
                     print("#### %s csv file %s_all.csv has been generated for %s dataset ####" %('test',category, category.upper()))
                     write_file(file,path)
             else:
-                with open(os.path.join(path_category,category+'_train.csv'), 'w', newline='') as file:
+                with open(os.path.join(base_path,category+'_train.csv'), 'w', newline='') as file:
                     print("#### %s csv file %s_all.csv has been generated for %s dataset ####" %('train',category,  category.upper()))
-                    write_file(file,path)
+                    write_file(file, path)
                     
 
 def write_file(file,path):
diff --git a/docker/flex-gpu/flex-gpu-docker-compose.yml b/docker/flex-gpu/flex-gpu-docker-compose.yml
deleted file mode 100644
index a280ced8b..000000000
--- a/docker/flex-gpu/flex-gpu-docker-compose.yml
+++ /dev/null
@@ -1,95 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-version: '3'
-services:
-  pytorch-resnet50v1-5-inference:
-    image: intel/image-recognition:pytorch-flex-gpu-resnet50v1-5-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${FLEX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${FLEX_PYT_BASE_TAG:-xpu-flex}
-      dockerfile: docker/flex-gpu/pytorch-resnet50v1-5-inference/pytorch-flex-series-resnet50v1-5-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  pytorch-ssd-mobilenet-inference:
-    image: intel/object-detection:pytorch-flex-gpu-ssd-mobilenet-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${FLEX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${FLEX_PYT_BASE_TAG:-xpu-flex}
-        VOC_LABELS_URL: ${VOC_LABELS_URL:-https://drive.google.com/uc?id=1q1sXhKIxniInw3WASnEDKYMqPMuiDDvc}   
-      dockerfile: docker/flex-gpu/pytorch-ssd-mobilenet-inference/pytorch-flex-series-ssd-mobilenet-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  pytorch-yolov4-inference:
-    image: intel/object-detection:pytorch-flex-gpu-yolov4-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${FLEX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${FLEX_PYT_BASE_TAG:-xpu-flex}   
-      dockerfile: docker/flex-gpu/pytorch-yolov4-inference/pytorch-flex-series-yolov4-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  tf-resnet50v1-5-inference:
-    image: intel/image-recognition:tf-flex-gpu-resnet50v1-5-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${FLEX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        BASE_TAG: ${FLEX_TF_BASE_TAG:-gpu}   
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
-      dockerfile: docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
-  tf-ssd-mobilenet-inference:
-    image: intel/object-detection:tf-flex-gpu-ssd-mobilenet-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${FLEX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        BASE_TAG: ${FLEX_TF_BASE_TAG:-gpu}
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/ssd_mobilenet_v1_int8_itex.pb}   
-      dockerfile: docker/flex-gpu/tf-ssd-mobilenet-inference/tf-flex-series-ssd-mobilenet-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
diff --git a/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_140.yml b/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_140.yml
deleted file mode 100644
index a373a8273..000000000
--- a/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_140.yml
+++ /dev/null
@@ -1,24 +0,0 @@
----
-pytorch-flex-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:pytorch-flex-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50V1_5 INT8 block Dummy batch inference on Flex 140
-    env_vars:
-      SCRIPT: quickstart/flex_multi_card_batch_inference.sh 
-      BATCH_SIZE: 256
-      NUM_ITERATIONS: 500
-      PRECISION: int8
-      DOCKER_ARGS: --cap-add=SYS_NICE --ipc=host --privileged --device=/dev/dri
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
-    volumes:
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
-  - test_name: ResNet50V1_5 INT8 block Dummy online inference on Flex 140
-    env_vars:
-      SCRIPT: quickstart/flex_multi_card_online_inference.sh 
-      BATCH_SIZE: 1
-      NUM_ITERATIONS: 5000
-      PRECISION: int8
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
-      DOCKER_ARGS: --cap-add=SYS_NICE --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
diff --git a/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_170.yml b/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_170.yml
deleted file mode 100644
index c85ede518..000000000
--- a/docker/flex-gpu/pytorch-resnet50v1-5-inference/tests_flex_170.yml
+++ /dev/null
@@ -1,14 +0,0 @@
----
-pytorch-flex-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:pytorch-flex-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50V1_5 INT8 block Dummy inference
-    env_vars:
-      SCRIPT: quickstart/inference_block_format.sh
-      BATCH_SIZE: 1024
-      NUM_ITERATIONS: 500
-      PRECISION: int8
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
-    volumes:
-      OUTPUT_DIR: /workspace/pytorch-flex-gpu-resnet50v1-5-inference
diff --git a/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_140.yml b/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_140.yml
deleted file mode 100644
index 4e92c2a16..000000000
--- a/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_140.yml
+++ /dev/null
@@ -1,28 +0,0 @@
----
-tf-flex-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:tf-flex-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50v1_5 INT8 batch inference on Flex 140
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: flex_series
-      BATCH_SIZE: 256
-      SCRIPT: quickstart/flex_multi_card_batch_inference.sh
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-      DOCKER_ARGS: --cap-add=SYS_NICE --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-  - test_name: ResNet50v1_5 INT8 online inference on Flex 140
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: flex_series
-      BATCH_SIZE: 1
-      SCRIPT: quickstart/flex_multi_card_online_inference.sh
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-      DOCKER_ARGS: --cap-add=SYS_NICE --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
diff --git a/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_170.yml b/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_170.yml
deleted file mode 100644
index 2736be062..000000000
--- a/docker/flex-gpu/tf-resnet50v1-5-inference/tests_flex_170.yml
+++ /dev/null
@@ -1,37 +0,0 @@
----
-tf-flex-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:tf-flex-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50v1_5 INT8 Batch inference
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: flex_series
-      SCRIPT: quickstart/batch_inference.sh
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-  - test_name: ResNet50v1_5 INT8 Online inference
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: flex_series
-      SCRIPT: quickstart/online_inference.sh
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-    volumes:
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-  - test_name: ResNet50v1_5 INT8 Accuracy inference
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: flex_series
-      SCRIPT: quickstart/accuracy.sh
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
-    volumes:
-      DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-      OUTPUT_DIR: /output/tf-flex-gpu-resnet50v1-5-inference
diff --git a/docker/max-gpu/docker-compose.yml b/docker/max-gpu/docker-compose.yml
index 3adbd29dc..2c77cf619 100644
--- a/docker/max-gpu/docker-compose.yml
+++ b/docker/max-gpu/docker-compose.yml
@@ -27,8 +27,8 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        MAX_PYT_BASE_IMAGE: ${MAX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        MAX_PYT_BASE_TAG: ${MAX_PYT_BASE_TAG:-xpu-max}
+        BASE_IMAGE: ${REGISTRY}/aiops/mlops
+        BASE_TAG: xpu-max
       dockerfile: docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
     command: >
       sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
@@ -54,8 +54,8 @@ services:
   tf-resnet50v1-5-inference:
     build:
       args:
-        MAX_TF_BASE_IMAGE: ${MAX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        MAX_TF_BASE_TAG: ${MAX_TF_BASE_TAG:-gpu-horovod}   
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-tensorflow}
+        BASE_TAG: ${BASE_TAG:-xpu}   
         INT8_MODEL_URL: ${INT8_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
         FP16_MODEL_URL: ${FP16_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
         FP32_MODEL_URL: ${FP32_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
@@ -72,15 +72,22 @@ services:
   tf-bert-large-inference:
     build:
       args:
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/fp32_bert_squad.pb}
+        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/bert_frozen_graph_fp32.pb}
       dockerfile: docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile
     extends: tf-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
   tf-bert-large-training:
     build:
-      args:
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
       dockerfile: docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
     extends: tf-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-training
-
+  pytorch-dlrm-inference:
+    build:
+      dockerfile: docker/max-gpu/pytorch-dlrm-inference/pytorch-max-series-dlrm-inference.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-max-gpu-dlrm-inference
+  pytorch-dlrm-training:
+    build:
+      dockerfile: docker/max-gpu/pytorch-dlrm-training/pytorch-max-series-dlrm-training.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-max-gpu-dlrm-training
diff --git a/docker/max-gpu/max-gpu-docker-compose.yml b/docker/max-gpu/max-gpu-docker-compose.yml
deleted file mode 100644
index 30490c633..000000000
--- a/docker/max-gpu/max-gpu-docker-compose.yml
+++ /dev/null
@@ -1,148 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-version: '3'
-services:
-  pytorch-resnet50v1-5-inference:
-    image: intel/image-recognition:pytorch-max-gpu-resnet50v1-5-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_PYT_BASE_IMAGE: ${MAX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        MAX_PYT_BASE_TAG: ${MAX_PYT_BASE_TAG:-xpu-max}
-      dockerfile: docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-
-  pytorch-resnet50v1-5-training:
-    image: intel/image-recognition:pytorch-max-gpu-resnet50v1-5-training
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_PYT_BASE_IMAGE: ${MAX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        MAX_PYT_BASE_TAG: ${MAX_PYT_BASE_TAG:-xpu-max}
-        VOC_LABELS_URL: ${VOC_LABELS_URL:-https://drive.google.com/uc?id=1q1sXhKIxniInw3WASnEDKYMqPMuiDDvc}   
-      dockerfile: docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-
-  pytorch-bert-large-inference:
-    image: intel/language-modeling:pytorch-max-gpu-bert-large-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_PYT_BASE_IMAGE: ${MAX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        MAX_PYT_BASE_TAG: ${MAX_PYT_BASE_TAG:-xpu-max}
-      dockerfile: docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-
-  pytorch-bert-large-training:
-    image: intel/language-modeling:pytorch-max-gpu-bert-large-training
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_PYT_BASE_IMAGE: ${MAX_PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        MAX_PYT_BASE_TAG: ${MAX_PYT_BASE_TAG:-xpu-max}
-      dockerfile: docker/max-gpu/pytorch-bert-large-training/pytorch-max-series-bert-large-training.Dockerfile
-    command: >
-      sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-
-  tf-resnet50v1-5-inference:
-    image: intel/image-recognition:tf-max-gpu-resnet50v1-5-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_TF_BASE_IMAGE: ${MAX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        MAX_TF_BASE_TAG: ${MAX_TF_BASE_TAG:-gpu-horovod}   
-        INT8_MODEL_URL: ${INT8_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
-        FP16_MODEL_URL: ${FP16_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
-        FP32_MODEL_URL: ${FP32_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
-      dockerfile: docker/max-gpu/tf-resnet50v1-5-inference/tf-max-series-resnet50v1-5-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
-
-  tf-resnet50v1-5-training:
-    image: intel/image-recognition:tf-max-gpu-resnet50v1-5-training
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_TF_BASE_IMAGE: ${MAX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        MAX_TF_BASE_TAG: ${MAX_TF_BASE_TAG:-gpu-horovod}   
-      dockerfile: docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
-      
-  tf-bert-large-inference:
-    image: intel/language-modeling:tf-max-gpu-bert-large-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_TF_BASE_IMAGE: ${MAX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        MAX_TF_BASE_TAG: ${MAX_TF_BASE_TAG:-gpu-horovod}   
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/fp32_bert_squad.pb}
-      dockerfile: docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
-
-  tf-bert-large-training:
-    image: intel/language-modeling:tf-max-gpu-bert-large-training
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        MAX_TF_BASE_IMAGE: ${MAX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        MAX_TF_BASE_TAG: ${MAX_TF_BASE_TAG:-gpu-horovod}    
-        MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
-      dockerfile: docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
-    command: >
-      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
-      
diff --git a/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile b/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
index 0453a1710..3057e0417 100644
--- a/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
+++ b/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG MAX_PYT_BASE_TAG="xpu-max"
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
 
-FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
+FROM ${BASE_IMAGE}:${BASE_TAG}
 
 WORKDIR /workspace/pytorch-max-series-bert-large-inference
 
diff --git a/docker/max-gpu/pytorch-bert-large-inference/tests.yaml b/docker/max-gpu/pytorch-bert-large-inference/tests.yaml
new file mode 100644
index 000000000..dc3f4bf85
--- /dev/null
+++ b/docker/max-gpu/pytorch-bert-large-inference/tests.yaml
@@ -0,0 +1,19 @@
+fp16-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+  cmd: quickstart/fp16_inference_plain_format.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: FP16
+    BATCH_SIZE: '64'
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+    DATASET_DIR: /dataset/bert_squad
+  volumes:
+    - src: /dataset/bert_squad
+      dst: /dataset/bert_squad
+    - src: /dataset/squad_large_finetuned_checkpoint
+      dst: /dataset/squad_large_finetuned_checkpoint
+
+      
diff --git a/docker/max-gpu/pytorch-bert-large-inference/tests.yml b/docker/max-gpu/pytorch-bert-large-inference/tests.yml
deleted file mode 100644
index 554cc55b7..000000000
--- a/docker/max-gpu/pytorch-bert-large-inference/tests.yml
+++ /dev/null
@@ -1,15 +0,0 @@
----
-pytorch-max-gpu-bert-large-inference:
-  image_name: intel/language-modeling:pytorch-max-gpu-bert-large-inference
-  tests:
-  - test_name: BERT Large FP16 Plain format inference
-    env_vars:
-      SCRIPT: quickstart/fp16_inference_plain_format.sh
-      Tile: 2
-      DATASET_DIR: /pytorch/squad_data
-      BERT_WEIGHT: /pytorch/squad_large_finetuned_checkpoint 
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /pytorch/squad_data
-      BERT_WEIGHT: /pytorch/squad_large_finetuned_checkpoint 
-      
diff --git a/docker/max-gpu/pytorch-dlrm-inference/pytorch-max-series-dlrm-inference.Dockerfile b/docker/max-gpu/pytorch-dlrm-inference/pytorch-max-series-dlrm-inference.Dockerfile
new file mode 100644
index 000000000..4a49494d6
--- /dev/null
+++ b/docker/max-gpu/pytorch-dlrm-inference/pytorch-max-series-dlrm-inference.Dockerfile
@@ -0,0 +1,51 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
+
+FROM ${BASE_IMAGE}:${BASE_TAG}
+
+WORKDIR /workspace/pytorch-max-series-dlrm-inference
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing numactl
+    
+RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
+RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    ca-certificates \
+    intel-oneapi-mpi-devel=2021.10.0-49371 \
+    intel-oneapi-ccl=2021.10.0-49084 \
+    && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY models/recommendation/pytorch/torchrec_dlrm/inference/gpu models/recommendation/pytorch/torchrec_dlrm/inference/gpu
+COPY quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_card_distributed_inference.sh quickstart/multi_card_distributed_inference.sh
+
+RUN cd models/recommendation/pytorch/torchrec_dlrm/inference/gpu && \
+    pip install -r requirements.txt && \
+    cd -
+    
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/max-gpu/pytorch-dlrm-inference/tests.yaml b/docker/max-gpu/pytorch-dlrm-inference/tests.yaml
new file mode 100644
index 000000000..96c65e2ea
--- /dev/null
+++ b/docker/max-gpu/pytorch-dlrm-inference/tests.yaml
@@ -0,0 +1,23 @@
+fp16-multi-card-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-max-gpu-dlrm-inference
+  cmd: quickstart/multi_card_distributed_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    GLOBAL_BATCH_SIZE: '65536' 
+    TOTAL_TRAINING_SAMPLES: '4195197692'
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    DATASET_DIR: /dataset/dlrm-v2
+    PRETRAINED_MODEL: /dlrm-v2_weights/model_weights
+  volumes:
+    - src: /dataset/dlrm-v2
+      dst: /dataset/dlrm-v2
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /dlrm-v2_weights/model_weights
+      dst: /dlrm-v2_weights/model_weights
+    
+
+      
diff --git a/docker/max-gpu/pytorch-dlrm-training/pytorch-max-series-dlrm-training.Dockerfile b/docker/max-gpu/pytorch-dlrm-training/pytorch-max-series-dlrm-training.Dockerfile
new file mode 100644
index 000000000..c651b1faa
--- /dev/null
+++ b/docker/max-gpu/pytorch-dlrm-training/pytorch-max-series-dlrm-training.Dockerfile
@@ -0,0 +1,52 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
+
+FROM ${BASE_IMAGE}:${BASE_TAG}
+
+WORKDIR /workspace/pytorch-max-series-dlrm-training
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing numactl 
+    
+RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
+RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    ca-certificates \
+    intel-oneapi-mpi-devel=2021.10.0-49371 \
+    intel-oneapi-ccl=2021.10.0-49084 \
+    && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY models/recommendation/pytorch/torchrec_dlrm/training/gpu models/recommendation/pytorch/torchrec_dlrm/training/gpu
+COPY quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_card_distributed_train.sh quickstart/multi_card_distributed_train.sh 
+
+RUN cd models/recommendation/pytorch/torchrec_dlrm/training/gpu && \
+    pip install -r requirements.txt && \
+    cd -
+RUN pip install -e git+https://github.com/mlperf/logging#egg=mlperf-logging
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/max-gpu/pytorch-dlrm-training/tests.yaml b/docker/max-gpu/pytorch-dlrm-training/tests.yaml
new file mode 100644
index 000000000..81790f192
--- /dev/null
+++ b/docker/max-gpu/pytorch-dlrm-training/tests.yaml
@@ -0,0 +1,19 @@
+bf16-multi-card-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-max-gpu-dlrm-training
+  cmd: quickstart/multi_card_distributed_train.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: bf16
+    GLOBAL_BATCH_SIZE: '65536' 
+    TOTAL_TRAINING_SAMPLES: '4195197692'
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    DATASET_DIR: /dataset/dlrm-v2
+  volumes:
+    - src: /dataset/dlrm-v2
+      dst: /dataset/dlrm-v2
+    - src: /dev/dri
+      dst: /dev/dri
+
+      
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile b/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
index 6eed070ae..c966852ae 100644
--- a/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
+++ b/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG MAX_PYT_BASE_TAG="xpu-max"
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
 
-FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
+FROM ${BASE_IMAGE}:${BASE_TAG}
 
 WORKDIR /workspace/pytorch-max-series-resnet50v1-5-inference
 COPY quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md README.md
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml b/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml
new file mode 100644
index 000000000..6745f1d71
--- /dev/null
+++ b/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml
@@ -0,0 +1,11 @@
+int8-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/inference_block_format.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: int8
+    NUM_ITERATIONS: '500'
+    BATCH_SIZE: '1024'
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yml b/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yml
deleted file mode 100644
index 499063e7b..000000000
--- a/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yml
+++ /dev/null
@@ -1,15 +0,0 @@
----
-pytorch-max-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:pytorch-max-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50V1_5 INT8 block Dummy inference
-    env_vars:
-      SCRIPT: quickstart/inference_block_format.sh
-      Tile: 2
-      BATCH_SIZE: 1024
-      NUM_ITERATIONS: 10
-      PRECISION: int8
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-      OUTPUT_DIR: /output/pytorch-max-gpu-resnet50v1-5-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-max-gpu-resnet50v1-5-inference
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile b/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
index 451a87f19..c1f30446f 100644
--- a/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
+++ b/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG MAX_PYT_BASE_TAG="xpu-max"
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
 
-FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
+FROM ${BASE_IMAGE}:${BASE_TAG}
 
 RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
 RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
diff --git a/docker/max-gpu/tf-bert-large-inference/tests.yaml b/docker/max-gpu/tf-bert-large-inference/tests.yaml
new file mode 100644
index 000000000..91a6e3b03
--- /dev/null
+++ b/docker/max-gpu/tf-bert-large-inference/tests.yaml
@@ -0,0 +1,162 @@
+fp32-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: benchmark
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+fp16-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: benchmark
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+bf16-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: benchmark
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: accuracy
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+fp16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: accuracy
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: accuracy
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+fp32-profile-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: profile
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+fp16-profile-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: profile
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
+bf16-profile-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-inference
+  cmd: quickstart/inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    PRETRAINED_DIR: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    SQUAD_DIR: /dataset/squad_dir
+    NUM_OAM: '4'
+    FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+    INFERENCE_MODE: profile
+  volumes:
+    - src: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+      dst: /dataset/pretrained_dir/wwm_uncased_L-24_H-1024_A-16
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
diff --git a/docker/max-gpu/tf-bert-large-inference/tests.yml b/docker/max-gpu/tf-bert-large-inference/tests.yml
deleted file mode 100644
index 4fba8ef7d..000000000
--- a/docker/max-gpu/tf-bert-large-inference/tests.yml
+++ /dev/null
@@ -1,33 +0,0 @@
----
-tf-max-gpu-bert-large-inference:
-  image_name: intel/language-modeling:tf-max-gpu-bert-large-inference
-  tests:
-  - test_name: BERT Large FP16 benchmark
-    env_vars:
-      PRECISION: fp16
-      Tile: 2
-      FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
-      PRETRAINED_DIR: /pytorch/wwm_uncased_L-24_H-1024_A-16
-      SQUAD_DIR: /pytorch/bert-large-inference/squad_dir
-      SCRIPT: quickstart/benchmark.sh
-      OUTPUT_DIR: /output/tf-max-gpu-bert-large-inference
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-max-gpu-bert-large-inference
-      PRETRAINED_DIR: /pytorch/wwm_uncased_L-24_H-1024_A-16
-      SQUAD_DIR: /pytorch/bert-large-inference/squad_dir
-  - test_name: BERT Large FP32 benchmark
-    env_vars:
-      PRECISION: fp32
-      
-      Tile: 2
-      FROZEN_GRAPH: /workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
-      PRETRAINED_DIR: /pytorch/wwm_uncased_L-24_H-1024_A-16
-      SQUAD_DIR: /pytorch/bert-large-inference/squad_dir
-      SCRIPT: quickstart/benchmark.sh
-      OUTPUT_DIR: /output/tf-max-gpu-bert-large-inference/fp32
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-max-gpu-bert-large-inference/fp32
-      PRETRAINED_DIR: /pytorch/bert-large-inference
-      SQUAD_DIR: /pytorch/bert-large-inference/squad_dir
diff --git a/docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile b/docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile
index 748f5ba11..9565e2f42 100644
--- a/docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile
+++ b/docker/max-gpu/tf-bert-large-inference/tf-max-series-bert-large-inference.Dockerfile
@@ -12,17 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ============================================================================
-#
-# THIS IS A GENERATED DOCKERFILE.
-#
-# This file was assembled from multiple pieces, whose use is documented
-# throughout. Please refer to the TensorFlow dockerfiles documentation
-# for more information.
 
-ARG MAX_TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG MAX_TF_BASE_TAG="gpu-horovod"
+ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG BASE_TAG="xpu"
 
-FROM ${MAX_TF_BASE_IMAGE}:${MAX_TF_BASE_TAG}
+FROM ${BASE_IMAGE}:${BASE_TAG}
 
 WORKDIR /workspace/tf-max-series-bert-large-inference
 
@@ -31,12 +25,9 @@ ARG MODEL_URL
 RUN mkdir -p frozen_graph && \
     wget ${MODEL_URL} -O frozen_graph/fp32_bert_squad.pb
 
-COPY benchmarks benchmarks
-COPY models models
-COPY quickstart/common quickstart/common
+COPY models/language_modeling/tensorflow/bert_large/inference/gpu models/language_modeling/tensorflow/bert_large/inference/gpu
 
-COPY quickstart/language_modeling/tensorflow/bert_large/inference/gpu/accuracy.sh quickstart/accuracy.sh
-COPY quickstart/language_modeling/tensorflow/bert_large/inference/gpu/benchmark.sh quickstart/benchmark.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/gpu/inference.sh quickstart/inference.sh
 COPY quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md quickstart/README.md
 
 COPY LICENSE licenses/LICENSE
diff --git a/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile b/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
index fb2e3c699..bab2ae609 100644
--- a/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
+++ b/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG BASE_TAG="gpu-horovod"
+ARG MAX_TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG MAX_TF_BASE_TAG="gpu-horovod"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${MAX_TF_BASE_IMAGE}:${MAX_TF_BASE_TAG}
 
 RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
 RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
diff --git a/docker/max-gpu/tf-resnet50v1-5-inference/tests.yaml b/docker/max-gpu/tf-resnet50v1-5-inference/tests.yaml
new file mode 100644
index 000000000..6a01ab1ef
--- /dev/null
+++ b/docker/max-gpu/tf-resnet50v1-5-inference/tests.yaml
@@ -0,0 +1,105 @@
+fp32-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/batch_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
+fp16-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/batch_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
+int8-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/batch_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    NUM_OAM: '4'
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+fp32-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/online_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
+fp16-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/online_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
+int8-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/online_inference.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/accuracy.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+  volumes:
+    - src: /tf_dataset/dataset/TF_Imagenet_FullData
+      dst: /tf_dataset/dataset/TF_Imagenet_FullData
+fp16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/accuracy.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+  volumes:
+    - src: /tf_dataset/dataset/TF_Imagenet_FullData
+      dst: /tf_dataset/dataset/TF_Imagenet_FullData
+int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-inference
+  cmd: quickstart/accuracy.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    GPU_TYPE: max_series
+    FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+  volumes:
+    - src: /tf_dataset/dataset/TF_Imagenet_FullData
+      dst: /tf_dataset/dataset/TF_Imagenet_FullData
diff --git a/docker/max-gpu/tf-resnet50v1-5-inference/tests.yml b/docker/max-gpu/tf-resnet50v1-5-inference/tests.yml
deleted file mode 100644
index 8d6ff7b65..000000000
--- a/docker/max-gpu/tf-resnet50v1-5-inference/tests.yml
+++ /dev/null
@@ -1,122 +0,0 @@
----
-tf-max-gpu-resnet50v1-5-inference:
-  image_name: intel/image-recognition:tf-max-gpu-resnet50v1-5-inference
-  tests:
-  - test_name: ResNet50v1_5 INT8 Batch inference
-    env_vars:
-      PRECISION: int8
-      Tile: 2
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
-      SCRIPT: quickstart/batch_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-  - test_name: ResNet50v1_5 INT8 Online inference
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/online_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-  - test_name: ResNet50v1_5 INT8 Accuracy inference
-    env_vars:
-      PRECISION: int8
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/accuracy.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/int8
-  - test_name: ResNet50v1_5 FP16 Batch inference
-    env_vars:
-      PRECISION: fp16
-      Tile: 2
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
-      SCRIPT: quickstart/batch_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-  - test_name: ResNet50v1_5 FP16 Online inference
-    env_vars:
-      PRECISION: fp16
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/online_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-  - test_name: ResNet50v1_5 FP16 Accuracy inference
-    env_vars:
-      PRECISION: fp16
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp16-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/accuracy.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp16
-  - test_name: ResNet50v1_5 FP32 Batch inference
-    env_vars:
-      PRECISION: fp32
-      Tile: 2
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
-      SCRIPT: quickstart/batch_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-  - test_name: ResNet50v1_5 FP32 Online inference
-    env_vars:
-      PRECISION: fp32
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/online_inference.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-  - test_name: ResNet50v1_5 FP32 Accuracy inference
-    env_vars:
-      PRECISION: fp32
-      GPU_TYPE: max_series
-      FROZEN_GRAPH: /workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-fp32-gpu.pb
-      Tile: 1
-      SCRIPT: quickstart/accuracy.sh
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      DOCKER_ARGS: --ipc=host --privileged --device=/dev/dri
-    volumes:
-      DATASET_DIR: /tf_dataset/datasets/TF_record
-      OUTPUT_DIR: /output/tf-max-gpu-resnet50v1-5-inference/fp32
-      
diff --git a/docker/max-gpu/tf-resnet50v1-5-inference/tf-max-series-resnet50v1-5-inference.Dockerfile b/docker/max-gpu/tf-resnet50v1-5-inference/tf-max-series-resnet50v1-5-inference.Dockerfile
index 12163d943..d67ac2cbb 100644
--- a/docker/max-gpu/tf-resnet50v1-5-inference/tf-max-series-resnet50v1-5-inference.Dockerfile
+++ b/docker/max-gpu/tf-resnet50v1-5-inference/tf-max-series-resnet50v1-5-inference.Dockerfile
@@ -19,13 +19,16 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG MAX_TF_BASE_TAG="gpu-horovod"
+ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG BASE_TAG="xpu"
 
-FROM ${MAX_TF_BASE_IMAGE}:${MAX_TF_BASE_TAG}
+FROM ${BASE_IMAGE}:${BASE_TAG}
 
 WORKDIR /workspace/tf-max-series-resnet50v1-5-inference
 
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl
+
 ARG INT8_MODEL_URL
 ARG FP16_MODEL_URL
 ARG FP32_MODEL_URL
diff --git a/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile b/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
index 2ad4db65b..738879d3c 100644
--- a/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
+++ b/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG MAX_TF_BASE_TAG="gpu-horovod"
+ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG MAX_PYT_BASE_TAG="gpu-horovod"
 
-FROM ${MAX_TF_BASE_IMAGE}:${MAX_TF_BASE_TAG}
+FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
 
 RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
 RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
diff --git a/docker/pyt-cpu/docker-compose.yml b/docker/pyt-cpu/docker-compose.yml
new file mode 100644
index 000000000..d3cb03b3e
--- /dev/null
+++ b/docker/pyt-cpu/docker-compose.yml
@@ -0,0 +1,215 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+version: '3'
+services:
+  spr-bert-large-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-spr-bert-large-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-maskrcnn-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-maskrcnn-inference/pytorch-spr-maskrcnn-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-maskrcnn-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-maskrcnn-training/pytorch-spr-maskrcnn-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-resnet50-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-resnet50-training/pytorch-spr-resnet50-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-resnet50-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-resnet50-inference/pytorch-spr-resnet50-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-ssd-resnet34-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-ssd-resnet34-inference/pytorch-spr-ssd-resnet34-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-ssd-resnet34-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-ssd-resnet34-training/pytorch-spr-ssd-resnet34-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-resnext-32x16d-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-dlrm-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-dlrm-inference/pytorch-spr-dlrm-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-dlrm-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-dlrm-training/pytorch-spr-dlrm-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-rnnt-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-rnnt-training/pytorch-spr-rnnt-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-rnnt-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-rnnt-inference/pytorch-spr-rnnt-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-distilbert-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-distilbert-inference/pytorch-spr-distilbert-inference.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
+  spr-bert-large-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
+      dockerfile: docker/pyt-cpu/spr-bert-large-training/pytorch-spr-bert-large-training.Dockerfile
+    command: >
+      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
diff --git a/docker/pyt-cpu/pyt-cpu-docker-compose.yml b/docker/pyt-cpu/pyt-cpu-docker-compose.yml
deleted file mode 100644
index aaef60bd8..000000000
--- a/docker/pyt-cpu/pyt-cpu-docker-compose.yml
+++ /dev/null
@@ -1,36 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-version: '3'
-services:
-  pytorch-spr-bert-large-inference:
-    image: intel/language-modeling:pytorch-spr-bert-large-inference
-    pull_policy: always
-    build:
-      context: ../../
-      args:
-        http_proxy: ${http_proxy}
-        https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        PYT_BASE_IMAGE: ${PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        PYT_BASE_TAG: ${PYT_BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
-    command: >
-      bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
diff --git a/docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile b/docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
index 89919ac0d..f3d2de58a 100644
--- a/docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
+++ b/docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
@@ -19,10 +19,10 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG PYT_BASE_TAG="2.0.0-pip-base"
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
 
-FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG} AS intel-extension-for-pytorch
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
 RUN apt-get update && \
     apt-get install --no-install-recommends --fix-missing -y \
     build-essential \
diff --git a/docker/pyt-cpu/spr-bert-large-inference/tests.yaml b/docker/pyt-cpu/spr-bert-large-inference/tests.yaml
new file mode 100644
index 000000000..aba0e2f02
--- /dev/null
+++ b/docker/pyt-cpu/spr-bert-large-inference/tests.yaml
@@ -0,0 +1,32 @@
+fp32-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-spr-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-spr-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
diff --git a/docker/pyt-cpu/spr-bert-large-inference/tests.yml b/docker/pyt-cpu/spr-bert-large-inference/tests.yml
deleted file mode 100644
index f3227b116..000000000
--- a/docker/pyt-cpu/spr-bert-large-inference/tests.yml
+++ /dev/null
@@ -1,214 +0,0 @@
----
-pytorch-spr-bert-large-inference:
-  image_name: intel/language-modeling:pytorch-spr-bert-large-inference
-  tests:
-  - test_name: BERT Large FP32 Online inference
-    env_vars:
-      PRECISION: fp32
-      SCRIPT: quickstart/run_multi_instance_realtime.sh
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/real_time
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large FP32 Throughput inference
-    env_vars:
-      PRECISION: fp32
-      SCRIPT: quickstart/run_multi_instance_throughput.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large FP32 Accuracy
-    env_vars:
-      PRECISION: fp32
-      SCRIPT: quickstart/run_accuracy.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/fp32/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large BFloat16 Online inference
-    env_vars:
-      PRECISION: bf16
-      SCRIPT: quickstart/run_multi_instance_realtime.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/real_time
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large BFloat16 Throughput inference
-    env_vars:
-      PRECISION: bf16
-      SCRIPT: quickstart/run_multi_instance_throughput.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large BFloat16 Accuracy
-    env_vars:
-      PRECISION: bf16
-      SCRIPT: quickstart/run_accuracy.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/accuracy
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/bf16/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large INT8 Online inference
-    env_vars:
-      PRECISION: int8
-      SCRIPT: quickstart/run_multi_instance_realtime.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/real_time
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large INT8 Throughput inference
-    env_vars:
-      PRECISION: int8
-      SCRIPT: quickstart/run_multi_instance_throughput.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/throughput
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large INT8 Accuracy
-    env_vars:
-      PRECISION: int8
-      SCRIPT: quickstart/run_accuracy.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/accuracy
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/int8/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-int8 Online inference
-    env_vars:
-      PRECISION: avx-int8
-      SCRIPT: quickstart/run_multi_instance_realtime.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-int8 Throughput inference
-    env_vars:
-      PRECISION: avx-int8
-      SCRIPT: quickstart/run_multi_instance_throughput.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/throughput
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-int8 Accuracy
-    env_vars:
-      PRECISION: avx-int8
-      SCRIPT: quickstart/run_accuracy.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/accuracy
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS: --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-int8/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-fp32 Online inference
-    env_vars:
-      PRECISION: avx-fp32
-      SCRIPT: quickstart/run_multi_instance_realtime.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/real_time
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-fp32 Throughput inference
-    env_vars:
-      PRECISION: avx-fp32
-      SCRIPT: quickstart/run_multi_instance_throughput.sh
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/throughput
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS:  --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/throughput
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
-  - test_name: BERT Large avx-fp32 Accuracy
-    env_vars:
-      PRECISION: avx-fp32
-      SCRIPT: quickstart/run_accuracy.sh
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/accuracy
-      INT8_CONFIG: /workspace/pytorch-spr-bert-large-inference/quickstart/configure.json
-      EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      FINETUNED_MODEL: /localdisk/sramakr1/bert_squad_model
-      DOCKER_ARGS: --privileged --init --shm-size 8G -w /workspace/pytorch-spr-bert-large-inference
-    volumes:
-      OUTPUT_DIR: /output/pytorch-spr-bert-large-inference/avx-fp32/accuracy
-      EVAL_DATA_FILE: /localdisk/sramakr1/eval_data_file/dev-v1.1.json
-      PRETRAINED_MODEL: /localdisk/sramakr1/bert_squad_model
diff --git a/docker/pyt-cpu/spr-bert-large-training/pytorch-spr-bert-large-training.Dockerfile b/docker/pyt-cpu/spr-bert-large-training/pytorch-spr-bert-large-training.Dockerfile
new file mode 100644
index 000000000..a4fd504aa
--- /dev/null
+++ b/docker/pyt-cpu/spr-bert-large-training/pytorch-spr-bert-large-training.Dockerfile
@@ -0,0 +1,83 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+WORKDIR /workspace/pytorch-spr-bert-large-training
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+COPY models/language_modeling/pytorch/bert_large/training models/language_modeling/pytorch/bert_large/training
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh quickstart/run_bert_pretrain_phase1.sh
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh quickstart/run_bert_pretrain_phase2.sh
+
+ARG TRANSFORMERS_COMMIT="v4.28.1"
+
+RUN pip install datasets==1.11.0 accelerate tfrecord intel-openmp faiss-cpu && \
+    apt-get install -y libopenblas-dev && \
+    cd quickstart && \
+    git clone https://github.com/huggingface/transformers.git && \
+    cd transformers && \
+    git checkout ${TRANSFORMERS_COMMIT} && \
+    git apply /workspace/pytorch-spr-bert-large-training/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+    python -m pip install --upgrade pip && \
+    pip uninstall transformers -y && \
+    pip install -e . && \
+    pip install h5py && \
+    mkdir -p /root/.local
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libtcmalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-bert-large-training/tests.yaml b/docker/pyt-cpu/spr-bert-large-training/tests.yaml
new file mode 100644
index 000000000..283517764
--- /dev/null
+++ b/docker/pyt-cpu/spr-bert-large-training/tests.yaml
@@ -0,0 +1,72 @@
+fp32-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-spr-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /localdisk/sramakr1/tfrecord_dir
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /localdisk/sramakr1/tfrecord_dir
+      dst: /localdisk/sramakr1/tfrecord_dir
+fp32-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-spr-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /localdisk/sramakr1/tfrecord_dir
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /localdisk/sramakr1/tfrecord_dir
+      dst: /localdisk/sramakr1/tfrecord_dir
+bf16-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-spr-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /localdisk/sramakr1/tfrecord_dir
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /localdisk/sramakr1/tfrecord_dir
+      dst: /localdisk/sramakr1/tfrecord_dir
+bf16-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-spr-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-spr-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /localdisk/sramakr1/tfrecord_dir
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /localdisk/sramakr1/tfrecord_dir
+      dst: /localdisk/sramakr1/tfrecord_dir
diff --git a/docker/pyt-cpu/spr-distilbert-inference/pytorch-spr-distilbert-inference.Dockerfile b/docker/pyt-cpu/spr-distilbert-inference/pytorch-spr-distilbert-inference.Dockerfile
new file mode 100644
index 000000000..99112b960
--- /dev/null
+++ b/docker/pyt-cpu/spr-distilbert-inference/pytorch-spr-distilbert-inference.Dockerfile
@@ -0,0 +1,88 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+WORKDIR /workspace/pytorch-spr-distilbert-inference
+
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json configure.json
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh  quickstart/run_accuracy.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh quickstart/do_calibration.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh quickstart/run_multi_instance_realtime.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh quickstart/run_multi_instance_throughput.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/download_model_dataset.sh quickstart/download_model_dataset.sh
+
+RUN cd quickstart && \
+    wget https://raw.githubusercontent.com/huggingface/datasets/2.0.0/metrics/accuracy/accuracy.py && \
+    git clone https://github.com/huggingface/transformers.git && \
+    cd transformers && \
+    git checkout v4.28.1 && \
+    pip install -e ./ && \
+    pip install -r examples/pytorch/language-modeling/requirements.txt && \
+    pip install tensorboard intel-openmp && \
+    mkdir -p /root/.local && \
+    pip install datasets scikit-learn scipy && \
+    apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    git-lfs && \
+    git lfs install
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libtcmalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-distilbert-inference/tests.yaml b/docker/pyt-cpu/spr-distilbert-inference/tests.yaml
new file mode 100644
index 000000000..67e8af512
--- /dev/null
+++ b/docker/pyt-cpu/spr-distilbert-inference/tests.yaml
@@ -0,0 +1,270 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp32-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh fp16
+  env:
+    PRECISION: fp16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh fp16
+  env:
+    PRECISION: fp16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp16-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh fp16
+  env:
+    PRECISION: fp16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh int8-bf16
+  env:
+    PRECISION: int8-bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh int8-bf16
+  env:
+    PRECISION: int8-fp16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh int8-bf16
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '1'
+    HF_DATASETS_OFFLINE: '1'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-spr-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-spr-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
diff --git a/docker/pyt-cpu/spr-dlrm-inference/pytorch-spr-dlrm-inference.Dockerfile b/docker/pyt-cpu/spr-dlrm-inference/pytorch-spr-dlrm-inference.Dockerfile
new file mode 100644
index 000000000..73a3a87fd
--- /dev/null
+++ b/docker/pyt-cpu/spr-dlrm-inference/pytorch-spr-dlrm-inference.Dockerfile
@@ -0,0 +1,70 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml intel-openmp
+
+WORKDIR /workspace/pytorch-spr-dlrm-inference
+
+COPY models/recommendation/pytorch/dlrm models/recommendation/pytorch/dlrm
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh quickstart/inference_performance.sh
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-spr-dlrm-inference/quickstart/requirements.txt
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-dlrm-inference/tests.yaml b/docker/pyt-cpu/spr-dlrm-inference/tests.yaml
new file mode 100644
index 000000000..04ee56d63
--- /dev/null
+++ b/docker/pyt-cpu/spr-dlrm-inference/tests.yaml
@@ -0,0 +1,112 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+int8-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+fp32-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+int8-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
diff --git a/docker/pyt-cpu/spr-dlrm-training/pytorch-spr-dlrm-training.Dockerfile b/docker/pyt-cpu/spr-dlrm-training/pytorch-spr-dlrm-training.Dockerfile
new file mode 100644
index 000000000..7b1db88f5
--- /dev/null
+++ b/docker/pyt-cpu/spr-dlrm-training/pytorch-spr-dlrm-training.Dockerfile
@@ -0,0 +1,68 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml intel-openmp
+
+WORKDIR /workspace/pytorch-spr-dlrm-training
+
+COPY models/recommendation/pytorch/dlrm models/recommendation/pytorch/dlrm
+COPY quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-spr-dlrm-training/quickstart/requirements.txt
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-dlrm-training/tests.yaml b/docker/pyt-cpu/spr-dlrm-training/tests.yaml
new file mode 100644
index 000000000..0b1c90d91
--- /dev/null
+++ b/docker/pyt-cpu/spr-dlrm-training/tests.yaml
@@ -0,0 +1,36 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: fp32
+    NUM_BATCH: '1500'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: bf16
+    NUM_BATCH: '1500'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: bf32
+    NUM_BATCH: '1500'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
diff --git a/docker/pyt-cpu/spr-maskrcnn-inference/pytorch-spr-maskrcnn-inference.Dockerfile b/docker/pyt-cpu/spr-maskrcnn-inference/pytorch-spr-maskrcnn-inference.Dockerfile
new file mode 100644
index 000000000..7bf287c61
--- /dev/null
+++ b/docker/pyt-cpu/spr-maskrcnn-inference/pytorch-spr-maskrcnn-inference.Dockerfile
@@ -0,0 +1,88 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN pip install intel-openmp 
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts defusedxml
+
+WORKDIR /workspace/pytorch-spr-maskrcnn-inference
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/common quickstart/common
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh quickstart/accuracy.sh 
+
+# RUN cd /workspace/pytorch-spr-maskrcnn-inference/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+#     pip install setuptools==58.2.0
+
+RUN apt-get update && apt-get install -y python3.10-dev && \
+    cd /workspace/pytorch-spr-maskrcnn-inference/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop && \
+    cd -
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libgl1 \
+    libglib2.0-0 \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-maskrcnn-inference/tests.yaml b/docker/pyt-cpu/spr-maskrcnn-inference/tests.yaml
new file mode 100644
index 000000000..00fb2ebbd
--- /dev/null
+++ b/docker/pyt-cpu/spr-maskrcnn-inference/tests.yaml
@@ -0,0 +1,84 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/accuracy.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  cmd: quickstart/accuracy.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
diff --git a/docker/pyt-cpu/spr-maskrcnn-training/pytorch-spr-maskrcnn-training.Dockerfile b/docker/pyt-cpu/spr-maskrcnn-training/pytorch-spr-maskrcnn-training.Dockerfile
new file mode 100644
index 000000000..aa3eb9418
--- /dev/null
+++ b/docker/pyt-cpu/spr-maskrcnn-training/pytorch-spr-maskrcnn-training.Dockerfile
@@ -0,0 +1,78 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml intel-openmp 
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+WORKDIR /workspace/pytorch-spr-maskrcnn-training
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh quickstart/training.sh
+
+RUN apt-get update && apt-get install -y python3.10-dev && \
+    cd /workspace/pytorch-spr-maskrcnn-training/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop 
+
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libgl1 \
+    libglib2.0-0 \
+    libegl1-mesa
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-maskrcnn-training/tests.yaml b/docker/pyt-cpu/spr-maskrcnn-training/tests.yaml
new file mode 100644
index 000000000..b194f3e6e
--- /dev/null
+++ b/docker/pyt-cpu/spr-maskrcnn-training/tests.yaml
@@ -0,0 +1,22 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-training
+  cmd: quickstart/training.sh fp32
+  env:
+    PRECISION: fp32
+    BATCH_SIZE: '56'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /localdisk/sharvils/data/coco
+  volumes:
+    - src: /localdisk/sharvils/data/coco
+      dst: /localdisk/sharvils/data/coco
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-training
+  cmd: quickstart/training.sh bf16
+  env:
+    PRECISION: bf16
+    BATCH_SIZE: '56'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /localdisk/sharvils/data/coco
+  volumes:
+    - src: /localdisk/sharvils/data/coco
+      dst: /localdisk/sharvils/data/coco
diff --git a/docker/pyt-cpu/spr-resnet50-inference/pytorch-spr-resnet50-inference.Dockerfile b/docker/pyt-cpu/spr-resnet50-inference/pytorch-spr-resnet50-inference.Dockerfile
new file mode 100644
index 000000000..cbebb807b
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnet50-inference/pytorch-spr-resnet50-inference.Dockerfile
@@ -0,0 +1,70 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml 
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+WORKDIR /workspace/pytorch-spr-resnet50-inference 
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-resnet50-inference/tests.yaml b/docker/pyt-cpu/spr-resnet50-inference/tests.yaml
new file mode 100644
index 000000000..811470586
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnet50-inference/tests.yaml
@@ -0,0 +1,132 @@
+fp32-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf16-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+int8-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+avx-int8-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+fp32-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf16-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+int8-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+avx-int8-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+fp32-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf16-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bfloat16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+int8-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+avx-int8-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
diff --git a/docker/pyt-cpu/spr-resnet50-training/pytorch-spr-resnet50-training.Dockerfile b/docker/pyt-cpu/spr-resnet50-training/pytorch-spr-resnet50-training.Dockerfile
new file mode 100644
index 000000000..9894d7d59
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnet50-training/pytorch-spr-resnet50-training.Dockerfile
@@ -0,0 +1,74 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml intel-openmp
+
+WORKDIR /workspace/pytorch-spr-resnet50-training 
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh quickstart/training.sh
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-resnet50-training/tests.yaml b/docker/pyt-cpu/spr-resnet50-training/tests.yaml
new file mode 100644
index 000000000..7799402f2
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnet50-training/tests.yaml
@@ -0,0 +1,22 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
diff --git a/docker/pyt-cpu/spr-resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile b/docker/pyt-cpu/spr-resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile
new file mode 100644
index 000000000..fe1f6cc6b
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile
@@ -0,0 +1,74 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ===========================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml
+
+WORKDIR /workspace/pytorch-spr-resnext-32x16d-inference 
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/common quickstart/common
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh quickstart//inference_throughput.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+ENV PATH="~/conda/bin:${PATH}"
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-resnext-32x16d-inference/tests.yaml b/docker/pyt-cpu/spr-resnext-32x16d-inference/tests.yaml
new file mode 100644
index 000000000..e2fb6dc70
--- /dev/null
+++ b/docker/pyt-cpu/spr-resnext-32x16d-inference/tests.yaml
@@ -0,0 +1,132 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
diff --git a/docker/pyt-cpu/spr-rnnt-inference/pytorch-spr-rnnt-inference.Dockerfile b/docker/pyt-cpu/spr-rnnt-inference/pytorch-spr-rnnt-inference.Dockerfile
new file mode 100644
index 000000000..b207d4678
--- /dev/null
+++ b/docker/pyt-cpu/spr-rnnt-inference/pytorch-spr-rnnt-inference.Dockerfile
@@ -0,0 +1,88 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+WORKDIR /workspace/pytorch-spr-rnnt-inference
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+COPY models/language_modeling/pytorch/rnnt/inference/cpu models/language_modeling/pytorch/rnnt/inference/cpu
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh 
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY  quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN cd /workspace/pytorch-spr-rnnt-inference/models/language_modeling/pytorch/rnnt/inference/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    pip install --force-reinstall soundfile && \
+    apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    libsndfile1 && \
+    apt-get update && apt-get install -y python3.10-dev && \
+    mkdir -p /root/.local && \
+    git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    python setup.py install
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-rnnt-inference/tests.yaml b/docker/pyt-cpu/spr-rnnt-inference/tests.yaml
new file mode 100644
index 000000000..fdefa2702
--- /dev/null
+++ b/docker/pyt-cpu/spr-rnnt-inference/tests.yaml
@@ -0,0 +1,84 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh fp16
+  shm-size: 8G
+  env:
+    PRECISION: fp16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_realtime.sh fp16
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp16-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh fp16
+  shm-size: 8G
+  env:
+    PRECISION: fp16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
diff --git a/docker/pyt-cpu/spr-rnnt-training/pytorch-spr-rnnt-training.Dockerfile b/docker/pyt-cpu/spr-rnnt-training/pytorch-spr-rnnt-training.Dockerfile
new file mode 100644
index 000000000..c46201a64
--- /dev/null
+++ b/docker/pyt-cpu/spr-rnnt-training/pytorch-spr-rnnt-training.Dockerfile
@@ -0,0 +1,90 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+WORKDIR /workspace/pytorch-spr-rnnt-training 
+
+COPY models/language_modeling/pytorch/rnnt/training/cpu models/language_modeling/pytorch/rnnt/training/cpu
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN cd /workspace/pytorch-spr-rnnt-training/models/language_modeling/pytorch/rnnt/training/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    pip install --force-reinstall soundfile && \
+    pip install protobuf==3.20.* && \
+    pip install --upgrade pip && \
+    pip install librosa sox && \
+    apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    libsndfile1 && \
+    mkdir -p /root/.local && \
+    git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    apt-get update && apt-get install -y python3.10-dev && \
+    python setup.py install
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-rnnt-training/tests.yaml b/docker/pyt-cpu/spr-rnnt-training/tests.yaml
new file mode 100644
index 000000000..f99eaaaa0
--- /dev/null
+++ b/docker/pyt-cpu/spr-rnnt-training/tests.yaml
@@ -0,0 +1,28 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-training
+  cmd: quickstart/training.sh fp32
+  shm-size: 8G
+  env: 
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset_training
+  volumes:
+    - src: /pytorch/rnnt_dataset_training
+      dst: /pytorch/rnnt_dataset_training
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-training
+  cmd: quickstart/training.sh bf16
+  shm-size: 8G
+  env: 
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset_training
+  volumes:
+    - src: /pytorch/rnnt_dataset_training
+      dst: /pytorch/rnnt_dataset_training
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
diff --git a/docker/pyt-cpu/spr-ssd-resnet34-inference/pytorch-spr-ssd-resnet34-inference.Dockerfile b/docker/pyt-cpu/spr-ssd-resnet34-inference/pytorch-spr-ssd-resnet34-inference.Dockerfile
new file mode 100644
index 000000000..1e8340604
--- /dev/null
+++ b/docker/pyt-cpu/spr-ssd-resnet34-inference/pytorch-spr-ssd-resnet34-inference.Dockerfile
@@ -0,0 +1,71 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml 
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+WORKDIR /workspace/pytorch-spr-ssd-resnet34-inference
+
+COPY models/object_detection/pytorch/ssd-resnet34/inference/cpu models/object_detection/pytorch/ssd-resnet34/inference/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+# ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so:/root/conda/envs/pytorch/lib/libiomp5.so:$LD_PRELOAD"
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-ssd-resnet34-inference/tests.yaml b/docker/pyt-cpu/spr-ssd-resnet34-inference/tests.yaml
new file mode 100644
index 000000000..5de779665
--- /dev/null
+++ b/docker/pyt-cpu/spr-ssd-resnet34-inference/tests.yaml
@@ -0,0 +1,156 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
diff --git a/docker/pyt-cpu/spr-ssd-resnet34-training/pytorch-spr-ssd-resnet34-training.Dockerfile b/docker/pyt-cpu/spr-ssd-resnet34-training/pytorch-spr-ssd-resnet34-training.Dockerfile
new file mode 100644
index 000000000..723702515
--- /dev/null
+++ b/docker/pyt-cpu/spr-ssd-resnet34-training/pytorch-spr-ssd-resnet34-training.Dockerfile
@@ -0,0 +1,83 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+# Build Torch Vision
+ARG TORCHVISION_VERSION="01c11a0564b8417561ae4c414fe659fc97476987"
+
+RUN git clone https://github.com/pytorch/vision && \
+    cd vision && \
+    git checkout ${TORCHVISION_VERSION} && \
+    python setup.py install
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/tcmalloc/lib/libjemalloc.so":$LD_PRELOAD
+
+RUN pip install matplotlib Pillow pycocotools pytz==2018.5 && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml
+
+WORKDIR /workspace/pytorch-spr-ssd-resnet34-training
+
+COPY models/object_detection/pytorch/ssd-resnet34/training/cpu models/object_detection/pytorch/ssd-resnet34/training/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh quickstart/throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh quickstart/accuracy.sh
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    libpng-dev \
+    libfreetype-dev && \
+    pip install --upgrade pip && \
+    pip install --no-cache-dir https://github.com/mlperf/logging/archive/9ea0afa.zip 
+
+ENV DNNL_MAX_CPU_ISA="AVX512_CORE_AMX"
+
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/spr-ssd-resnet34-training/tests.yaml b/docker/pyt-cpu/spr-ssd-resnet34-training/tests.yaml
new file mode 100644
index 000000000..cd01f197b
--- /dev/null
+++ b/docker/pyt-cpu/spr-ssd-resnet34-training/tests.yaml
@@ -0,0 +1,56 @@
+fp32-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+fp32-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf16-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf16-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
diff --git a/docker/requirements.txt b/docker/requirements.txt
deleted file mode 100644
index 1f963a59d..000000000
--- a/docker/requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-python-slugify
-pyyaml
-
diff --git a/docker/test_runner.py b/docker/test_runner.py
deleted file mode 100755
index 31ed3b7bc..000000000
--- a/docker/test_runner.py
+++ /dev/null
@@ -1,83 +0,0 @@
-
-import argparse
-import yaml
-import subprocess
-from slugify import slugify
-import os
-from collections import namedtuple
-
-TestResult = namedtuple('TestResult', 'test_name status log_file docker_cmd')
-
-def run_command(command, log_file_path):
-    with open(log_file_path, "w") as log_file:
-        process = subprocess.Popen(command.split(), stdout=log_file, stderr=log_file)
-        process.wait()
-    status = "SUCCESS" if process.returncode == 0 else "FAILED"
-    return status
-
-def main(tests_yaml, logs_dir):
-    print(tests_yaml, logs_dir)
-    with open(tests_yaml, "r") as stream:
-        try:
-            workloads = yaml.safe_load(stream)
-        except yaml.YAMLError as exc:
-            print(exc)
-
-    results = {}
-    for workload_name, workload_value in workloads.items():
-        print("workload_name: ", workload_name)
-        print(" image_name: ", workload_value['image_name'])
-        image_name = workload_value['image_name']
-        test_results = []
-        for test in workload_value['tests']:
-            test_name = slugify(test['test_name'])
-            log_file = os.path.join(logs_dir,
-                                    slugify(test['test_name'])+".log")
-            env_vars = test['env_vars'] #its list not dict
-            volumes = test['volumes']
-
-            docker_cmd = "docker run --rm " + env_vars['DOCKER_ARGS']
-            docker_cmd += " --env http_proxy={}".format(os.environ['http_proxy'])
-            docker_cmd += " --env https_proxy={}".format(os.environ['http_proxy'])
-            docker_cmd += " --env no_proxy={}".format(os.environ['no_proxy']) 
-            for key, value in env_vars.items():
-                docker_cmd += " --env {}={}".format(key,value)
-            #TODO: Bug - what if volume directory is does not exists, docker test will
-            #      eventually fail but test status will be success as docker cmd
-            #      will return success
-            for key, value in volumes.items():
-                docker_cmd += " --env {}={}".format(key, value)
-            for key, value in volumes.items():
-                docker_cmd += " --volume {}:{}".format(value, value)
-            docker_cmd += " " + image_name
-            docker_cmd += " /bin/bash " + env_vars['SCRIPT']
-
-            #print("docker cmd: ", docker_cmd)
-            status = run_command(docker_cmd, log_file)
-            test_results.append(TestResult(test_name, status, log_file, docker_cmd))
-            print("# ",slugify(test['test_name']), " - ", status, " - ", log_file)
-        results[workload_name] = test_results
-    # TODO: summerize all results
-    # with open(os.path.join(logs_dir, 'results.txt'), 'w') as f:
-    #     f.write("---- Test results ----\n")
-    #     for workload, test_results in results.items():
-    #         f.write(F"\n{workload} \n")
-    #         for test in test_results:
-    #             f.write(F"  {test.test_name} - {test.status}  - { test.log_file}  - {test.docker_cmd}\n")
-
-if __name__ == "__main__":
-    print("---- Test Runner --")
-    parser = argparse.ArgumentParser(description="Test runner execute tests run command and capture logs")
-
-    parser.add_argument("--tests-yaml", type=str, dest="tests_yaml", required=True,
-                        help="Yaml file specifying workload image and its tests")
-    parser.add_argument("--logs-dir", dest="logs_dir",
-                        default="./logs", type=str,
-                        help="Path to logs directory")
-
-    args = parser.parse_args()
-    os.makedirs(os.path.abspath(args.logs_dir),
-                           exist_ok=True)
-
-    main(os.path.abspath(args.tests_yaml), 
-         os.path.abspath(args.logs_dir))
diff --git a/docker/tf-cpu/.actions.json b/docker/tf-cpu/.actions.json
new file mode 100644
index 000000000..990b77193
--- /dev/null
+++ b/docker/tf-cpu/.actions.json
@@ -0,0 +1,4 @@
+{
+    "VARIATIONS": ["NONE"],
+    "experimental": [true]
+}
diff --git a/docker/dataset/dataset-docker-compose.yml b/docker/tf-cpu/docker-compose.yml
similarity index 70%
rename from docker/dataset/dataset-docker-compose.yml
rename to docker/tf-cpu/docker-compose.yml
index 22c6442b8..be6d9e302 100644
--- a/docker/dataset/dataset-docker-compose.yml
+++ b/docker/tf-cpu/docker-compose.yml
@@ -19,9 +19,9 @@
 #
 
 version: '3'
-services:
-  preprocess-coco-val:
-    image: intel/object-detection:tf-1.15.2-preprocess-coco-val
+services: 
+  tf-resnet50v1-5-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-resnet50v1-5-inference
     pull_policy: always
     build:
       context: ../../
@@ -29,9 +29,8 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        TF_MODELS_BRANCH: 7a9934df2afdf95be9405b4e9f1f2480d748dc40
-        BASE_IMAGE: ${TENSORFLOW_IMAGE:-intel/intel-optimized-tensorflow}
-        BASE_TAG: ${TENSORFLOW_TAG:-1.15.2}
-      dockerfile: docker/dataset/preprocess-coco-val/intel-tf-object-detection-preprocess-coco-val.Dockerfile
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile
     command: >
       sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
diff --git a/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml b/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml
new file mode 100644
index 000000000..7c2ac92c6
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml
@@ -0,0 +1,224 @@
+ResNet50 v1.5 BFloat32 accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
+    BATCH_SIZE: '100'
+    PRECISION: bfloat32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
+    dst: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
+ResNet50 v1.5 BFloat16 accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
+    BATCH_SIZE: '100'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
+    dst: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
+ResNet50 v1.5 fp32 accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
+    BATCH_SIZE: '100'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
+    dst: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
+ResNet50 v1.5 int8 accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
+    BATCH_SIZE: '100'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
+    dst: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
+ResNet50 v1.5 bfloat32 realtime inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
+    BATCH_SIZE: '1'
+    PRECISION: bfloat32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
+    dst: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
+ResNet50 v1.5 bfloat16 realtime inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
+    dst: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
+ResNet50 v1.5 fp32 realtime inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
+    dst: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
+ResNet50 v1.5 int8 realtime inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_realtime
+    BATCH_SIZE: '1'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/int8_realtime
+    dst: /output/tf-spr-resnet50v1-5-inference/int8_realtime
+ResNet50 v1.5 bfloat32 throughput inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
+    BATCH_SIZE: '448'
+    PRECISION: bfloat32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
+    dst: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
+ResNet50 v1.5 bfloat16 throughput inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
+    BATCH_SIZE: '448'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
+    dst: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
+ResNet50 v1.5 fp32 throughput inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
+    BATCH_SIZE: '448'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
+    dst: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
+ResNet50 v1.5 int8 throughput inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_throughput
+    BATCH_SIZE: '448'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/int8_throughput
+    dst: /output/tf-spr-resnet50v1-5-inference/int8_throughput
+ResNet50 v1.5 bfloat16 weightsharing inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
+    dst: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
+ResNet50 v1.5 int8 weightsharing inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
+    BATCH_SIZE: '1'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
+    dst: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
diff --git a/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile
new file mode 100644
index 000000000..92f4bcc8d
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile
@@ -0,0 +1,47 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-spr-resnet50v1-5-inference 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y ca-certificates numactl wget 
+
+RUN mkdir -p pretrained_model && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bias_resnet50.pb -O pretrained_model/bias_resnet50.pb && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bf16_resnet50_v1.pb -O pretrained_model/bf16_resnet50_v1.pb && \
+    wget https://zenodo.org/record/2535873/files/resnet50_v1.pb -O pretrained_model/resnet50_v1.pb
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh quickstart/inference_realtime_weightsharing.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docs/README.md b/docs/README.md
index 0d975d4dd..174223074 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -3,7 +3,7 @@
 ## Best Practices
 
 * Intel® Optimization for TensorFlow\*:
-    * [TensorFlow and the Model Zoo in the Intel® oneAPI AI Analytics Toolkit](/docs/general/tensorflow/AIKit.md)
+    * [TensorFlow and the Reference models in the Intel® oneAPI AI Tools](/docs/general/tensorflow/AITools.md)
     * [Installation Guide](https://www.intel.com/content/www/us/en/developer/articles/guide/optimization-for-tensorflow-installation-guide.html)
     * [General Best Practices](/docs/general/tensorflow/GeneralBestPractices.md)
 * Intel® Optimization for Tensorflow Serving\*:
diff --git a/docs/general/Driver_Setup.md b/docs/general/Driver_Setup.md
new file mode 100644
index 000000000..b79d96a8f
--- /dev/null
+++ b/docs/general/Driver_Setup.md
@@ -0,0 +1,15 @@
+# Driver Setup Instructions 
+
+This document provides the resources to perform driver installation and verification steps for Intel® Flex and Max Series GPUs.
+
+## Install Required Drivers
+
+Based on the Operating system requirements, follow the instructions given in the [link](https://dgpu-docs.intel.com/driver/installation.html#introduction) to install Kernel Mode Drivers(KMD) and User Mode Drivers(UMD). 
+
+## Verify Driver Installation
+
+Once drivers are installed, refer to the sections in the [link](https://dgpu-docs.intel.com/driver/installation.html#verify-install) to verify if drivers are installed successfully. 
+
+## Other resources
+
+Refer to the GPU driver releases [here](https://dgpu-docs.intel.com/releases/index.html).
diff --git a/docs/general/MAX_DEVCATALOG.md b/docs/general/MAX_DEVCATALOG.md
index 7a7e1b408..8e2ac6776 100644
--- a/docs/general/MAX_DEVCATALOG.md
+++ b/docs/general/MAX_DEVCATALOG.md
@@ -6,8 +6,8 @@ This document provides links to step-by-step instructions on how to leverage Mod
 
 | AI Framework                 | Extension            | Documentation |
 | -----------------------------| ------------- | ----------------- |
-| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/xpu-master/docker/README.md) |
-| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/r1.1/docker/README.md)|
+| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/v2.0.110%2Bxpu/docker/README.md) |
+| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/v2.13.0.0/docker/README.md)|
 
 ## Optimized Workloads
 
@@ -15,11 +15,16 @@ The table below provides links to run each workload in a docker container. The c
 
 | Model                            | Framework                  | Mode | Precisions | Dataset |
 | ----------------------------|     ---------- | ------------------- | ------------ | ------------ |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Inference](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Training](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/DEVCATALOG.md) | BF16 |  [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Inference](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md) | FP16 | [SQuAD](https://github.com/IntelAI/models/blob/master/datasets/bert_data/README.md#inference) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Training](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md) | BF16 | [MLCommons](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md#datasets) |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow| [Inference](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8,FP16,FP32 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [Training](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/DEVCATALOG.md) |  BF16 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Inference](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md) | FP16,FP32 | [SQuAD ](https://github.com/IntelAI/models/blob/master/datasets/bert_data/README.md#inference) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Training](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/tensorflow/bert_large/training/gpu/DEVCATALOG.md) | BF16 | Dummy dataset |
+| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8 | [ImageNet 2012](https://github.com/IntelAI/models/tree/v3.0.0/datasets/imagenet/README.md) |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md) | FP16 | [SQuAD](https://github.com/IntelAI/models/blob/v3.0.0/datasets/bert_data/README.md#inference) |
+| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md) | FP16 | [Criteo](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf) |
+| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Training](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md) | BF16 | [Criteo](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf) | 
+| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow| [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8,FP16,FP32 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md) | FP16,FP32 | [SQuAD ](https://github.com/IntelAI/models/blob/v3.0.0/datasets/bert_data/README.md#inference) |
+| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/DEVCATALOG.md) |  BF16 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
+| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/DEVCATALOG.md) | BF16 |  [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/language_modeling/tensorflow/bert_large/training/gpu/DEVCATALOG.md) | BF16 | Dummy dataset |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md) | BF16 | [MLCommons](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md#datasets) |
+
+
+**Note**: ResNet50v1.5 and BERT-Large training workloads are supported on older Intel® Extension for TensorFlow* v2.12 and Intel® Extension for PyTorch* 1.13.100+xpu versions. The other models in the list are validated on Intel® Extension for TensorFlow* v2.13 and Intel® Extension for PyTorch* 2.0.100+xpu versions.
diff --git a/docs/general/Windows.md b/docs/general/Windows.md
index 791cbab3e..02032186b 100644
--- a/docs/general/Windows.md
+++ b/docs/general/Windows.md
@@ -1,4 +1,4 @@
-# Using Model Zoo for Intel® Architecture on Windows Systems
+# Using Intel® AI Reference Models on Windows Systems
 
 ## Prerequisites for running on bare metal
 
diff --git a/docs/general/tensorflow/AIKit.md b/docs/general/tensorflow/AITools.md
similarity index 67%
rename from docs/general/tensorflow/AIKit.md
rename to docs/general/tensorflow/AITools.md
index fcda5555c..23fef320a 100644
--- a/docs/general/tensorflow/AIKit.md
+++ b/docs/general/tensorflow/AITools.md
@@ -1,13 +1,13 @@
-# Using the Model Zoo in the Intel® oneAPI AI Analytics Toolkit
+# Using Intel® AI Reference Models in AI Tools
 
-The Model Zoo is bundled as part of the
-[Intel® oneAPI AI Analytics Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html) (AI Kit).
-Follow the instructions below to get your environment setup with AI Kit to run
+The Intel® AI Reference Models repository source code is bundled as part of the
+[Intel® oneAPI AI Tools](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html) (AI Tools).
+Follow the instructions below to get your environment setup with AI Tools to run
 TensorFlow models.
 
-## Install AI Kit
+## Install AI Tools
 
-Use the link below for instructions on how to install AI Kit or run the AI Kit
+Use the link below for instructions on how to install AI Tools or run the AI Tools
 docker container:
 [https://www.intel.com/content/www/us/en/docs/oneapi-ai-analytics-toolkit/get-started-guide-linux/2023-1/overview.html](https://www.intel.com/content/www/us/en/docs/oneapi-ai-analytics-toolkit/get-started-guide-linux/2023-1/overview.html)
 
@@ -44,26 +44,18 @@ oneDNN optimizations are enabled (this should print `oneDNN optimizations enable
 python -c "from tensorflow.python.util import _pywrap_util_port; print('oneDNN optimizations enabled:', _pywrap_util_port.IsMklEnabled())"
 ```
 
-## Navigate to the Model Zoo
+## Navigate to the AI Reference Models
 
-Navigate to the Intel Model Zoo source directory. It's located in your oneapi installation path,
-typically `/opt/intel/oneapi/modelzoo`. You can view the available Model Zoo release versions
-for the Intel AI Analytics toolkit:
+Navigate to the Intel AI Reference Models source directory. It's located in your oneapi installation path, typically `/opt/intel/oneapi/ai_reference_models`.
 
+Then browse to the `models` directory to run the model:
 ```
-ls /opt/intel/oneapi/modelzoo
-2.4.0  latest
+cd /opt/intel/oneapi/ai_reference_models/models
 ```
 
-Then browse to the `models` directory in your preferred Intel Model Zoo release version location
-to run the model:
-```
-cd /opt/intel/oneapi/modelzoo/latest/models
-```
-
-You can use this version of the zoo to run models instead of cloning the repository
-from git. The [benchmarks README](/benchmarks/README.md) has a list of the
+You can use this version of the AI Reference Models to run models instead of cloning the repository
+from GitHub. The [README](/benchmarks/../README.md) has a list of the
 available models and links to their instructions. Note that individual models
 may require additional dependencies to be installed. You should follow the
-bare metal installation instructions in those documents to run in the AI Kit
-TF conda environment.
+bare metal installation instructions in those documents to run in the AI Tools
+TensorFlow conda environment.
diff --git a/docs/notebooks/README.md b/docs/notebooks/README.md
index f081803c9..3d920a56c 100644
--- a/docs/notebooks/README.md
+++ b/docs/notebooks/README.md
@@ -9,4 +9,4 @@ The contributor is responsible for maintaining their notebooks.
 | [perf_analysis](/docs/notebooks/perf_analysis) | Compare performance between stock and Intel Tensorflow for several models  | [louie-tsai](https://github.com/louie-tsai), [ZhuoweiSi](https://github.com/ZhuoweiSi), [yinghu5](https://github.com/yinghu5), [charulatha27](https://github.com/charulatha27)|
 | [ObjectDetection.ipynb](ObjectDetection.ipynb) | Visualize and benchmark the predictions of an object detection model served by TensorFlow Serving  | [mhbuehler](https://github.com/mhbuehler) |
 | [low_precision_optimization](/docs/notebooks/low_precision_optimization) | Quantize a model with Intel® Neural Compressor in Model Zoo for Intel(R) Architecture, and analyze the performance before and after quantization  | [NeoZhangJianyu](https://github.com/NeoZhangJianyu) |
-| [transfer_learning](/docs/notebooks/transfer_learning) | Demonstrates transfer learning using Intel TensorFlow and Intel Extension for PyTorch with the Model Zoo for Intel Architecture and other public model repositories | [dmsuehir](https://github.com/dmsuehir), [mhbuehler](https://github.com/mhbuehler) |
+| [transfer_learning](/docs/notebooks/transfer_learning) | Demonstrates transfer learning using Intel TensorFlow and Intel Extension for PyTorch with the Model Zoo for Intel Architecture and other public model repositories | [ashahba](https://github.com/ashahba), [dmsuehir](https://github.com/dmsuehir), [mhbuehler](https://github.com/mhbuehler), [okhleif-IL](https://github.com/okhleif-IL), [HarshaRamayanam](https://github.com/HarshaRamayanam), [etcylfleet](https://github.com/etcylfleet) |
diff --git a/docs/notebooks/transfer_learning/README.md b/docs/notebooks/transfer_learning/README.md
index c14f46766..03875c4d7 100644
--- a/docs/notebooks/transfer_learning/README.md
+++ b/docs/notebooks/transfer_learning/README.md
@@ -1,22 +1,29 @@
 # Transfer Learning Notebooks
 
-This directory has Jupyter notebooks that demonstrate transfer learning with
-models from the Model Zoo for Intel® Architecture and other public model repositories using Intel® Optimization for TensorFlow*
-and [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch).
+## Environment setup and running the notebooks
+
+Use the [setup instructions](setup.md) to install the dependencies required to run the notebooks.
+
+This directory has Jupyter notebooks that demonstrate transfer learning.
+All of the notebooks use models from public model repositories
+and leverage optimized libraries [ Intel® Optimization for TensorFlow](https://pypi.org/project/intel-tensorflow/)
+and [Intel® Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch).
 
 ## Natural Language Processing
 
-| Notebook | Framework | Description |
-| ---------| ----------|-------------|
-| [BERT Classifier fine tuning using the Model Zoo for Intel® Architecture](/docs/notebooks/transfer_learning/bert_classifier_fine_tuning/) | TensorFlow | Fine tunes BERT base from the Model Zoo for Intel® Architecture using the IMDb movie review dataset, then quantizes the saved model using the [Intel® Neural Compressor](https://github.com/intel/neural-compressor) |
-| [BERT SQuAD fine tuning with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert/) | TensorFlow | Demonstrates BERT fine tuning using scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) and the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook allows for selecting a BERT large or BERT base model from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-| [BERT Binary Text Classification with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert) | TensorFlow | Demonstrates BERT binary text classification fine tuning using the [IMDb movie review dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews) from [TensorFlow Datasets](https://www.tensorflow.org/datasets) or a custom dataset. The notebook allows for selecting a BERT encoder (BERT large, BERT base, or small BERT) to use along with a preprocessor from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-| [Text Classifier fine tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/pytorch_text_classification) | PyTorch | Demonstrates fine tuning [Hugging Face models](https://huggingface.co/models) to do sentiment analysis using the [IMDb movie review dataset from Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset with [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch) |
+| Notebook | Domain: Use Case | Framework| Description |
+| ---------| ---------|----------|-------------|
+| [BERT SQuAD fine tuning with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert/question_answering) | NLP: Question Answering | TensorFlow | Demonstrates BERT fine tuning using scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) and the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook allows for selecting a BERT large or BERT base model from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
+| [BERT Text Classification with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert/text_classification) | NLP: Text Classification | TensorFlow | Demonstrates BERT binary text classification fine tuning using the [IMDb movie review dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews) and multiclass text classification fine tuning using the [AG News datasets](https://www.tensorflow.org/datasets/catalog/ag_news_subset) from [TensorFlow Datasets](https://www.tensorflow.org/datasets) or a custom dataset (for binary classification). The notebook allows for selecting a BERT encoder (BERT large, BERT base, or small BERT) to use along with a preprocessor from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
+| [Text Classifier fine tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/pytorch_text_classification) | NLP: Text Classification | PyTorch |Demonstrates fine tuning [Hugging Face models](https://huggingface.co/models) to do sentiment analysis using the [IMDb movie review dataset from Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset with [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch) |
+| [LLM instruction tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/pytorch_text_generation) | NLP: Text Generation | PyTorch | Demonstrates instruction tuning [Hugging Face Causal Language Models](https://huggingface.co/docs/transformers/tasks/language_modeling) to do text generation using [Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset |
+
 
 ## Computer Vision
 
 | Notebook | Framework | Description |
 | ---------| ----------|-------------|
-| [Transfer Learning for Image Classification with TF Hub](/docs/notebooks/transfer_learning/tf_image_classification) | TensorFlow | Demonstrates transfer learning with multiple [TF Hub](https://tfhub.dev) image classifiers, TF datasets, and custom image datasets |
-| [Transfer Learning for Image Classification with PyTorch & torchvision](/docs/notebooks/transfer_learning/pytorch_image_classification) | PyTorch | Demonstrates transfer learning with multiple [torchvision](https://pytorch.org/vision/stable/index.html) image classification models, torchvision datasets, and custom datasets |
-| [Transfer Learning for Object Detection with PyTorch & torchvision](/docs/notebooks/transfer_learning/pytorch_object_detection) | PyTorch | Demonstrates transfer learning with multiple [torchvision](https://pytorch.org/vision/stable/index.html) object detection models, a public image dataset, and a customized torchvision dataset |
+| [Image Classification with TF Hub](/docs/notebooks/transfer_learning/tf_image_classification) | CV: Image Classification | TensorFlow | Demonstrates transfer learning with multiple [TF Hub](https://tfhub.dev) image classifiers, TF datasets, and custom image datasets |
+| [Image Classification with PyTorch & Torchvision](/docs/notebooks/transfer_learning/pytorch_image_classification) | CV: Image Classification | PyTorch | Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) image classification models, Torchvision datasets, and custom datasets |
+| [Image Classification with Hugging Face](/docs/notebooks/transfer_learning/huggingface_image_classification) | CV: Image Classification | PyTorch, TensorFlow | Demonstrates transfer learning with [Hugging Face models](https://huggingface.co/models) for image classification using either TensorFlow or PyTorch as a backend. |
+| [Object Detection with PyTorch & Torchvision](/docs/notebooks/transfer_learning/pytorch_object_detection) | CV: Object Detection | PyTorch |Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) object detection models, a public image dataset, and a customized Torchvision dataset |
diff --git a/docs/notebooks/transfer_learning/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb
new file mode 100644
index 000000000..41f73cba5
--- /dev/null
+++ b/docs/notebooks/transfer_learning/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb
@@ -0,0 +1,614 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f35836c9",
+   "metadata": {},
+   "source": [
+    "# Transfer Learning for Image Classification\n",
+    "\n",
+    "This notebook uses [ViT](https://huggingface.co/google/vit-base-patch16-224-in21k) classifier model from 🤗 model hub that was originally trained using [ImageNet](https://image-net.org) and does transfer learning with [Food101](https://huggingface.co/datasets/food101) dataset from 🤗 Datasets.\n",
+    "The notebook performs the following steps:\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
+    "2. [Load the Food101 dataset](#2.-Load-the-Food101-dataset)\n",
+    "3. [Preprocess the dataset](#3.-Preprocess-the-dataset)\n",
+    "4. [Transfer Learning](#4.-Transfer-Learning)\n",
+    "5. [Predict on test subset](#5.-Predict-on-test-subset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d4572dea",
+   "metadata": {},
+   "source": [
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the [README.md](/notebooks/README.md) to setup a 🤗 transformers environment with all the dependencies required to run the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6ebba765",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# General\n",
+    "# -------\n",
+    "import os\n",
+    "\n",
+    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"2\"\n",
+    "\n",
+    "import numpy as np\n",
+    "from PIL import Image\n",
+    "\n",
+    "\n",
+    "# Huggingface\n",
+    "# -----------\n",
+    "from datasets import load_dataset\n",
+    "from transformers import (\n",
+    "    AutoImageProcessor,\n",
+    "    DefaultDataCollator,\n",
+    "    AutoModelForImageClassification,\n",
+    "    TFAutoModelForImageClassification,\n",
+    "    TrainingArguments,\n",
+    "    Trainer,\n",
+    "    create_optimizer,\n",
+    "    pipeline\n",
+    ")\n",
+    "from transformers.keras_callbacks import KerasMetricCallback\n",
+    "import evaluate\n",
+    "\n",
+    "\n",
+    "# PyTorch\n",
+    "# -------\n",
+    "import torch\n",
+    "import torchvision.transforms as T\n",
+    "\n",
+    "\n",
+    "# TensorFlow\n",
+    "# ----------\n",
+    "import tensorflow as tf\n",
+    "from tensorflow import keras\n",
+    "from tensorflow.keras import layers"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "27af490e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Specify the the parent directory for the custom or Torchvision dataset\n",
+    "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
+    "    \n",
+    "# Specify a directory for output\n",
+    "output_directory = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
+    "\n",
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e4281a5",
+   "metadata": {},
+   "source": [
+    "### (Optional) Optimized CPU platform\n",
+    "\n",
+    "This is an optional step to optimize training time on Intel® Architecture. It uses certain environment variables with recommended values for optimized model training."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cb6aa15d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "\n",
+    "lscpu_output = subprocess.check_output(['lscpu'], stderr=subprocess.STDOUT)\n",
+    "if isinstance(lscpu_output, bytes):\n",
+    "    lscpu_output = lscpu_output.decode('utf-8')\n",
+    "    \n",
+    "cpu_info = lscpu_output.split('\\n')\n",
+    "\n",
+    "for line in cpu_info:\n",
+    "    if line.find('Core(s) per socket') == 0:\n",
+    "        num_cores_per_socket = int(line.split(\":\")[1].strip())\n",
+    "    elif line.find('Socket(s)') == 0:\n",
+    "        num_cpu_sockets = int(line.split(\":\")[1].strip())\n",
+    "\n",
+    "os.environ['OMP_NUM_THREADS'] = str(num_cores_per_socket * num_cpu_sockets)\n",
+    "os.environ['KMP_BLOCKTIME'] = '0'\n",
+    "os.environ['KMP_AFFINITY'] = \"granularity=fine,compact,1,0\"\n",
+    "os.environ['TF_NUM_INTRAOP_THREADS'] = str(num_cores_per_socket)\n",
+    "os.environ['TF_NUM_INTEROP_THREADS'] = str(num_cpu_sockets)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ecb2e91f",
+   "metadata": {},
+   "source": [
+    "## 2. Load the Food101 dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a7aabc83",
+   "metadata": {},
+   "source": [
+    "**Note:** In this notebook, we will load a subset of 5000 train samples from Food101 dataset. You can modify the `split` arg in the `load_dataset()` as you need."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1ff09cb7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load a subset of Food101 dataset\n",
+    "dataset = load_dataset('food101', split='train[:5000]', cache_dir=dataset_directory)\n",
+    "\n",
+    "# Split the dataset for training and evaluation\n",
+    "dataset = dataset.train_test_split(test_size=0.2)\n",
+    "\n",
+    "# Define variables to hold labels and their mappings\n",
+    "labels = dataset[\"train\"].features[\"label\"].names\n",
+    "label2id, id2label = dict(), dict()\n",
+    "for i, label in enumerate(labels):\n",
+    "    label2id[label] = str(i)\n",
+    "    id2label[str(i)] = label\n",
+    "\n",
+    "# Assign different variables for different frameworks\n",
+    "dataset_for_pyt = dataset\n",
+    "dataset_for_tf = dataset\n",
+    "\n",
+    "print(dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f7618f24",
+   "metadata": {},
+   "source": [
+    "### Inspect the dataset\n",
+    "\n",
+    "Select a random image from the dataset and see how it actually represented in the dataset object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "937e0b1b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "\n",
+    "select_num = random.randint(0, len(labels) - 1)\n",
+    "\n",
+    "img_dict = dataset['train'][select_num]\n",
+    "\n",
+    "print(img_dict)\n",
+    "print(id2label[str(img_dict['label'])])\n",
+    "img_dict['image']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56777018",
+   "metadata": {},
+   "source": [
+    "# 3. Preprocess the dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1969b791",
+   "metadata": {},
+   "source": [
+    "Run below cell which gets the matching image preprocessor for chosen model. This step is common for both PyTorch and TensorFlow frameworks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f980a80a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_name = \"google/vit-base-patch16-224-in21k\"\n",
+    "image_processor = AutoImageProcessor.from_pretrained(model_name)\n",
+    "\n",
+    "image_processor"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c2d69eaf",
+   "metadata": {},
+   "source": [
+    "### Option A: PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a700b7b3",
+   "metadata": {},
+   "source": [
+    "If using PyTorch as a backend, use the torch transforms to apply preprocessing to the datasets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1bc4c7d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "size = (\n",
+    "    image_processor.size[\"shortest_edge\"]\n",
+    "    if \"shortest_edge\" in image_processor.size\n",
+    "    else (image_processor.size[\"height\"], image_processor.size[\"width\"])\n",
+    ")\n",
+    "\n",
+    "# Define your transforms\n",
+    "_transforms = T.Compose([\n",
+    "    T.RandomResizedCrop(size), \n",
+    "    T.ToTensor(),\n",
+    "    T.Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\n",
+    "])\n",
+    "\n",
+    "# Apply the transforms to the dataset\n",
+    "def transforms(examples):\n",
+    "    examples[\"pixel_values\"] = [_transforms(img.convert(\"RGB\")) for img in examples[\"image\"]]\n",
+    "    del examples[\"image\"]\n",
+    "    return examples\n",
+    "\n",
+    "dataset_for_pyt = dataset_for_pyt.with_transform(transforms)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a445cee2",
+   "metadata": {},
+   "source": [
+    "Skip to next step [4. Transfer Learning](#4.-Transfer-Learning) to continue using PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7180f562",
+   "metadata": {},
+   "source": [
+    "### Option B: TensorFlow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c14e92e7",
+   "metadata": {},
+   "source": [
+    "If using TensorFlow as a backend, use the keras layers to apply preprocessing"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "97690ba1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "size = (image_processor.size[\"height\"], image_processor.size[\"width\"])\n",
+    "\n",
+    "# Define your keras layers for preprocessing\n",
+    "train_data_augmentation = keras.Sequential(\n",
+    "    [\n",
+    "        layers.RandomCrop(size[0], size[1]),\n",
+    "        layers.Rescaling(scale=1.0 / 127.5, offset=-1),\n",
+    "        layers.RandomFlip(\"horizontal\"),\n",
+    "        layers.RandomRotation(factor=0.02),\n",
+    "        layers.RandomZoom(height_factor=0.2, width_factor=0.2),\n",
+    "    ],\n",
+    "    name=\"train_data_augmentation\",\n",
+    ")\n",
+    "\n",
+    "val_data_augmentation = keras.Sequential(\n",
+    "    [\n",
+    "        layers.CenterCrop(size[0], size[1]),\n",
+    "        layers.Rescaling(scale=1.0 / 127.5, offset=-1),\n",
+    "    ],\n",
+    "    name=\"val_data_augmentation\",\n",
+    ")\n",
+    "\n",
+    "# Define helper functions to apply preprocessing layers\n",
+    "def convert_to_tf_tensor(image: Image):\n",
+    "    np_image = np.array(image)\n",
+    "    tf_image = tf.convert_to_tensor(np_image)\n",
+    "    # `expand_dims()` is used to add a batch dimension since\n",
+    "    # the TF augmentation layers operates on batched inputs.\n",
+    "    return tf.expand_dims(tf_image, 0)\n",
+    "\n",
+    "\n",
+    "def preprocess_train(example_batch):\n",
+    "    \"\"\"Apply train_transforms across a batch.\"\"\"\n",
+    "    images = [\n",
+    "        train_data_augmentation(convert_to_tf_tensor(image.convert(\"RGB\"))) for image in example_batch[\"image\"]\n",
+    "    ]\n",
+    "    example_batch[\"pixel_values\"] = [tf.transpose(tf.squeeze(image)) for image in images]\n",
+    "    return example_batch\n",
+    "\n",
+    "\n",
+    "def preprocess_val(example_batch):\n",
+    "    \"\"\"Apply val_transforms across a batch.\"\"\"\n",
+    "    images = [\n",
+    "        val_data_augmentation(convert_to_tf_tensor(image.convert(\"RGB\"))) for image in example_batch[\"image\"]\n",
+    "    ]\n",
+    "    example_batch[\"pixel_values\"] = [tf.transpose(tf.squeeze(image)) for image in images]\n",
+    "    return example_batch\n",
+    "\n",
+    "# Set the helper methods to the dataset(s)\n",
+    "dataset_for_tf[\"train\"].set_transform(preprocess_train)\n",
+    "dataset_for_tf[\"test\"].set_transform(preprocess_val)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2cb36e88",
+   "metadata": {},
+   "source": [
+    "## 4. Transfer Learning"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79938eff",
+   "metadata": {},
+   "source": [
+    "### Option A: PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e64f480",
+   "metadata": {},
+   "source": [
+    "If using PyTorch as a backend, get the model from 🤗 Auto... class and use 🤗 Trainer to train the model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d5c7923e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = AutoModelForImageClassification.from_pretrained(\n",
+    "    model_name,\n",
+    "    num_labels=len(labels),\n",
+    "    id2label=id2label,\n",
+    "    label2id=label2id,\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Define a function to calculate accuracy\n",
+    "accuracy = evaluate.load(\"accuracy\")\n",
+    "\n",
+    "def compute_metrics(eval_pred):\n",
+    "    predictions, labels = eval_pred\n",
+    "    predictions = np.argmax(predictions, axis=1)\n",
+    "    return accuracy.compute(predictions=predictions, references=labels)\n",
+    "\n",
+    "# Define training args for the Trainer\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir=output_directory,\n",
+    "    remove_unused_columns=False,\n",
+    "    evaluation_strategy=\"epoch\",\n",
+    "    save_strategy=\"epoch\",\n",
+    "    learning_rate=5e-5,\n",
+    "    per_device_train_batch_size=16,\n",
+    "    gradient_accumulation_steps=4,\n",
+    "    per_device_eval_batch_size=16,\n",
+    "    num_train_epochs=3,\n",
+    "    warmup_ratio=0.1,\n",
+    "    logging_steps=10,\n",
+    "    load_best_model_at_end=True,\n",
+    "    metric_for_best_model=\"accuracy\"\n",
+    ")\n",
+    "\n",
+    "# Define the Trainer class\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    data_collator=DefaultDataCollator(),\n",
+    "    train_dataset=dataset_for_pyt[\"train\"],\n",
+    "    eval_dataset=dataset_for_pyt[\"test\"],\n",
+    "    tokenizer=image_processor,\n",
+    "    compute_metrics=compute_metrics,\n",
+    ")\n",
+    "\n",
+    "# Train the model\n",
+    "model.train()  # Puts the model in training mode\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bcbfd409",
+   "metadata": {},
+   "source": [
+    "### Option B: TensorFlow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "503ef0d3",
+   "metadata": {},
+   "source": [
+    "If using TensorFlow as a backend, get the model from 🤗 TFAuto... class and use the TensorFlow's `fit()` method to train the model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bdac4b46",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "model = TFAutoModelForImageClassification.from_pretrained(\n",
+    "    model_name,\n",
+    "    id2label=id2label,\n",
+    "    label2id=label2id,\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Convert 🤗 Dataset to tf.data.Dataset\n",
+    "batch_size = 16\n",
+    "tf_train_dataset = dataset_for_tf[\"train\"].to_tf_dataset(\n",
+    "    columns=\"pixel_values\", label_cols=\"label\", shuffle=True, batch_size=batch_size,\n",
+    "    collate_fn=DefaultDataCollator(return_tensors=\"tf\")\n",
+    ")\n",
+    "\n",
+    "# converting our test dataset to tf.data.Dataset\n",
+    "tf_eval_dataset = dataset_for_tf[\"test\"].to_tf_dataset(\n",
+    "    columns=\"pixel_values\", label_cols=\"label\", shuffle=True, batch_size=batch_size,\n",
+    "    collate_fn=DefaultDataCollator(return_tensors=\"tf\")\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Define a function to calculate accuracy\n",
+    "accuracy = evaluate.load(\"accuracy\")\n",
+    "\n",
+    "def compute_metrics(eval_pred):\n",
+    "    predictions, labels = eval_pred\n",
+    "    predictions = np.argmax(predictions, axis=1)\n",
+    "    return accuracy.compute(predictions=predictions, references=labels)\n",
+    "\n",
+    "\n",
+    "# Create optimizer\n",
+    "num_epochs = 2\n",
+    "num_train_steps = len(dataset[\"train\"]) * num_epochs\n",
+    "learning_rate = 3e-5\n",
+    "weight_decay_rate = 0.01\n",
+    "\n",
+    "optimizer, lr_schedule = create_optimizer(\n",
+    "    init_lr=learning_rate,\n",
+    "    num_train_steps=num_train_steps,\n",
+    "    weight_decay_rate=weight_decay_rate,\n",
+    "    num_warmup_steps=0,\n",
+    ")\n",
+    "\n",
+    "# Define loss and callbacks\n",
+    "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
+    "callbacks = [KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_eval_dataset)]\n",
+    "\n",
+    "\n",
+    "# Compile the model\n",
+    "model.compile(optimizer=optimizer, loss=loss)\n",
+    "\n",
+    "# Train the model\n",
+    "model.fit(tf_train_dataset, validation_data=tf_eval_dataset, epochs=num_epochs, callbacks=callbacks)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f7e79e82",
+   "metadata": {},
+   "source": [
+    "## 5. Predict on test subset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bf4ae89a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tqdm import tqdm\n",
+    "import random\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "def get_predictions(image, model, framework):\n",
+    "    model_input = image_processor(\n",
+    "        image,\n",
+    "        return_tensors = 'tf' if framework == 'tensorflow' else 'pt'\n",
+    "    )\n",
+    "    if framework == 'tensorflow':\n",
+    "        logits = model(**model_input).logits\n",
+    "        predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])\n",
+    "    if framework == 'pytorch':\n",
+    "        with torch.no_grad():\n",
+    "            logits = model(**model_input).logits\n",
+    "        predicted_class_id = logits.argmax(-1).item()\n",
+    "    return predicted_class_id\n",
+    "\n",
+    "selected_image_indices = random.sample(range(0, dataset['test'].num_rows), 30)\n",
+    "selected_images = []\n",
+    "true_labels = []\n",
+    "predicted_labels = []\n",
+    "\n",
+    "for s in tqdm(selected_image_indices):\n",
+    "    img = dataset['test'][s]['image']\n",
+    "    label = dataset['test'][s]['label']\n",
+    "    selected_images.append(img)\n",
+    "    true_labels.append(label)\n",
+    "    predicted_labels.append(get_predictions(img, model, 'pytorch' if isinstance(model, torch.nn.Module) else 'tensorflow'))\n",
+    "    \n",
+    "# Visualize the predictions\n",
+    "plt.figure(figsize=(16,16))\n",
+    "plt.subplots_adjust(hspace=0.5)\n",
+    "for n in range(30):\n",
+    "    plt.subplot(6,5,n+1)\n",
+    "    plt.imshow(selected_images[n])\n",
+    "    correct_prediction = true_labels[n] == predicted_labels[n]\n",
+    "    color = \"darkgreen\" if correct_prediction else \"crimson\"\n",
+    "    true_label_name = model.config.id2label[str(true_labels[n])]\n",
+    "    predicted_label_name = model.config.id2label[str(predicted_labels[n])]\n",
+    "    title = predicted_label_name if correct_prediction else \"{}\\n({})\".format(predicted_label_name, true_label_name) \n",
+    "    plt.title(title, fontsize=14, color=color)\n",
+    "    plt.axis('off')\n",
+    "_ = plt.suptitle(\"ImageNet predictions\", fontsize=16)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "261a915b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/notebooks/transfer_learning/huggingface_image_classification/README.md b/docs/notebooks/transfer_learning/huggingface_image_classification/README.md
new file mode 100644
index 000000000..7de7e155e
--- /dev/null
+++ b/docs/notebooks/transfer_learning/huggingface_image_classification/README.md
@@ -0,0 +1,26 @@
+# Transfer Learning for Image Classification using HuggingFace
+
+This notebook uses [ViT](https://huggingface.co/google/vit-base-patch16-224-in21k) classifier model from 🤗 model hub that was originally trained using [ImageNet](https://image-net.org) and does transfer learning with [Food101](https://huggingface.co/datasets/food101) dataset from 🤗 Datasets.
+
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+2. Load the Food101 dataset
+3. Preprocess the dataset
+4. Transfer Learning (Pytorch/TensorFlow)
+5. Predict on test subset
+
+## Running the notebook
+
+To run the notebook, follow the instructions in `setup.md`.
+   
+## References
+
+Dataset citations:
+```
+@inproceedings{bossard14,
+  title = {Food-101 -- Mining Discriminative Components with Random Forests},
+  author = {Bossard, Lukas and Guillaumin, Matthieu and Van Gool, Luc},
+  booktitle = {European Conference on Computer Vision},
+  year = {2014}
+}
+```
diff --git a/docs/notebooks/transfer_learning/plot_utils.py b/docs/notebooks/transfer_learning/plot_utils.py
new file mode 100644
index 000000000..3d043bde4
--- /dev/null
+++ b/docs/notebooks/transfer_learning/plot_utils.py
@@ -0,0 +1,54 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import matplotlib.pyplot as plt
+
+
+def plot_curves(history, checkpoint_dir):
+    """
+    Reads a pickle file and plots accuracy and loss curves
+
+    :param history: Pickle file
+    :return: None
+    """
+    if not history:
+        raise FileNotFoundError("The pickle file {} does not exist".format(history))
+
+    acc = history['acc']
+    val_acc = history['val_acc']
+    loss = history['loss']
+    val_loss = history['val_loss']
+    plt.figure(figsize=(7, 7))
+    plt.subplot(2, 1, 1)
+    plt.plot(acc, label='Training Accuracy')
+    plt.plot(val_acc, label='Validation Accuracy')
+    plt.legend(loc='lower right')
+    plt.ylabel('Accuracy')
+    plt.title('Training and Validation Accuracy')
+
+    plt.subplot(2, 1, 2)
+    plt.plot(loss, label='Training Loss')
+    plt.plot(val_loss, label='Validation Loss')
+    plt.legend(loc='upper right')
+    plt.ylabel('Cross Entropy')
+    plt.title('Training and Validation Loss')
+    plt.xlabel('epoch')
+    if not os.path.exists(os.path.join(checkpoint_dir, 'train_val_plot.png')):
+        print("Saving plot in checkpoint_dir:", checkpoint_dir)
+        plt.savefig(os.path.join(checkpoint_dir, 'train_val_plot.png'))
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
index 6811074ff..524e17935 100644
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
@@ -1,13 +1,14 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
-   "id": "10088148",
+   "id": "67b1e912",
    "metadata": {},
    "source": [
     "# Transfer Learning for Image Classification\n",
     "\n",
-    "This notebook uses image classification models from [torchvision](https://pytorch.org/vision/stable/index.html) that were originally trained using [ImageNet](https://image-net.org/) and does transfer learning with a torchvision dataset or your own raw images.\n",
+    "This notebook uses image classification models from [Torchvision](https://pytorch.org/vision/stable/index.html) that were originally trained using [ImageNet](https://image-net.org/) and does transfer learning with a Torchvision dataset or your own raw images.\n",
     "\n",
     "The notebook performs the following steps:\n",
     "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
@@ -20,16 +21,18 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d20a1725",
+   "id": "939a7f98",
    "metadata": {},
    "source": [
-    "## 1. Import dependencies and setup parameters"
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the [README.md](/notebooks/README.md) to setup a PyTorch environment with all the dependencies required to run the notebook."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "40ec5e03",
+   "id": "25f98c12",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -44,6 +47,8 @@
     "from PIL import Image\n",
     "from pydoc import locate\n",
     "import warnings\n",
+    "import wget\n",
+    "from zipfile import ZipFile\n",
     "\n",
     "import intel_extension_for_pytorch as ipex\n",
     "import matplotlib.pyplot as plt\n",
@@ -59,14 +64,14 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "4198eedf",
+   "id": "e42b6e79",
    "metadata": {},
    "outputs": [],
    "source": [
     "# Specify a model from the list above\n",
     "model_name = \"efficientnet_b0\"\n",
     "\n",
-    "# Specify the the parent directory for the custom or torchvision dataset\n",
+    "# Specify the the parent directory for the custom or Torchvision dataset\n",
     "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
     "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
     "    \n",
@@ -75,13 +80,16 @@
     "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
     "\n",
     "# Batch size\n",
-    "batch_size = 32"
+    "batch_size = 32\n",
+    "\n",
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b4df6c73",
+   "id": "00f85b73",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -94,7 +102,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cffbd78",
+   "id": "1aaecdff",
    "metadata": {},
    "source": [
     "## 2. Prepare the dataset"
@@ -102,7 +110,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db1969d0",
+   "id": "98db4b34",
    "metadata": {},
    "source": [
     "Define transforms for data resizing and augmentation. The normalization means and standard deviations `[0.485, 0.456, 0.406], [0.229, 0.224, 0.225]` are specific to torchvision image classification models and are explained in the [documentation](https://pytorch.org/vision/stable/models.html#classification)."
@@ -111,7 +119,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "50b7b3b8",
+   "id": "429a90ea",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -130,34 +138,36 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
-   "id": "3895b6c0",
+   "id": "7cd4df40",
    "metadata": {},
    "source": [
-    "### Option A: Use a torchvision dataset"
+    "### Option A: Use a Torchvision dataset"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
-   "id": "0bfe54f6",
+   "id": "f0b814cb",
    "metadata": {},
    "source": [
-    "To use a torchvision dataset, load from the torchvision.datasets library, applying transforms for image augmentation, normalization, and resizing. This example uses the Food101 dataset from the [torchvision datasets for image classification](https://pytorch.org/vision/stable/datasets.html#image-classification), but you can choose from a variety of options. If the dataset is not found in the dataset directory it is downloaded. Subsequent runs will reuse the already downloaded dataset.\n",
+    "To use a Torchvision dataset, load from the Torchvision.datasets library, applying transforms for image augmentation, normalization, and resizing. This example uses the Food101 dataset from the [Torchvision datasets for image classification](https://pytorch.org/vision/stable/datasets.html#image-classification), but you can choose from a variety of options. If the dataset is not found in the dataset directory it is downloaded. Subsequent runs will reuse the already downloaded dataset.\n",
     "\n",
-    "Note: Some torchvision datasets use a `train=True/False` argument and others have a `split=\"train\"/\"test\"` convention. See the torchvision documentation to see how to specify the subset you want to use."
+    "Note: Some Torchvision datasets use a `train=True/False` argument and others have a `split=\"train\"/\"test\"` convention. See the Torchvision documentation to see how to specify the subset you want to use."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d93a068f",
+   "id": "537d83e1",
    "metadata": {},
    "outputs": [],
    "source": [
     "dataset = torchvision.datasets.Food101(dataset_directory, split='train',\n",
-    "                                            transform=get_transform(True), download=True)\n",
+    "                                       transform=get_transform(True), download=True)\n",
     "dataset_test = torchvision.datasets.Food101(dataset_directory, split='test',\n",
-    "                                                 transform=get_transform(False), download=True)   \n",
+    "                                            transform=get_transform(False), download=True)   \n",
     "class_names = dataset.classes\n",
     "\n",
     "print('Training data size: {}'.format(len(dataset)))\n",
@@ -166,7 +176,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77bd9055",
+   "id": "93bb716b",
    "metadata": {},
    "source": [
     "Now skip ahead to the [Predict using the original model](#3.-Predict-using-the-original-model) section."
@@ -174,7 +184,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "967ead90",
+   "id": "43f9656f",
    "metadata": {},
    "source": [
     "### Option B: Use a downloaded or custom dataset"
@@ -182,7 +192,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "238882c7",
+   "id": "f235fa0f",
    "metadata": {},
    "source": [
     "To use your own image dataset for transfer learning with the rest of this notebook, format your images as `.jpg` files and save them in folders named after the classes that you want the model to predict. To provide a working example using the correct layout, we will download and extract a flower species dataset. After downloading and extracting, you will have the following  subdirectories in your dataset directory. Each species subfolder will contain numerous `.jpg` files:\n",
@@ -203,7 +213,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7fca47b9",
+   "id": "29105a9e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -214,22 +224,26 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "60edf347",
+   "id": "13e96b9e",
    "metadata": {},
    "outputs": [],
    "source": [
     "# Only run this if you want to use the example flowers dataset\n",
     "if not os.path.exists(dataset_subdir):\n",
-    "    os.mkdir(dataset_subdir)\n",
-    "    !apt-get update && apt-get -qq install curl\n",
-    "    !wget https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\n",
-    "    !tar xvf flower_photos.tgz --directory $dataset_directory"
+    "    os.makedirs(dataset_subdir)\n",
+    "    dataset_url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
+    "    \n",
+    "    # Download and extract the tar\n",
+    "    zip_file = wget.download(dataset_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
+    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
+    "        zipfile.extractall(path=dataset_directory)\n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b1db045f",
+   "id": "a5482b8a",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -241,7 +255,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e65519c3",
+   "id": "e791a208",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -255,7 +269,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "22c2f194",
+   "id": "dbc2c043",
    "metadata": {},
    "source": [
     "## 3. Predict using the original model"
@@ -264,7 +278,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6efd0e1e",
+   "id": "a89f78d4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -276,17 +290,18 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3993d17a",
+   "id": "be4fa7fe",
    "metadata": {},
    "outputs": [],
    "source": [
     "# Get the ImageNet labels for displaying with the predictions\n",
     "imagenet_classes = []\n",
-    "labels_file = 'https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt'\n",
-    "if not os.path.exists('ImageNetLabels.txt'):\n",
-    "    !wget $labels_file\n",
+    "labels_file_url = 'https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt'\n",
+    "labels_file_path = os.path.join(dataset_directory, os.path.basename(labels_file_url))\n",
+    "if not os.path.exists(labels_file_url):\n",
+    "    wget.download(labels_file_url, dataset_directory)\n",
     "\n",
-    "with open('ImageNetLabels.txt') as f:\n",
+    "with open(labels_file_path) as f:\n",
     "    imagenet_labels = f.readlines()\n",
     "    imagenet_classes = [l.strip() for l in imagenet_labels]"
    ]
@@ -294,7 +309,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e6f5723e",
+   "id": "7444f838",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -313,7 +328,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "44a55db1",
+   "id": "0e893dcf",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -328,7 +343,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "52814d0f",
+   "id": "bfb66dc7",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -343,7 +358,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ec75ed87",
+   "id": "1d9409af",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -366,7 +381,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8bb4408f",
+   "id": "eba7d039",
    "metadata": {},
    "source": [
     "## 4. Transfer learning"
@@ -374,7 +389,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f8a827e9",
+   "id": "cc029ef3",
    "metadata": {},
    "source": [
     "Replace the pretrained head of the network with a new layer based on the number of classes in our dataset. Train the model using the new dataset for the specified number of epochs."
@@ -383,7 +398,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1b215888",
+   "id": "02f0eee5",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -399,7 +414,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "72fc7ca5",
+   "id": "41c2ee68",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -493,7 +508,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9c34072e",
+   "id": "f8b754b4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -509,7 +524,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "764d884a",
+   "id": "2a2c8e8b",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -519,7 +534,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "78aae3cd",
+   "id": "f22b2493",
    "metadata": {},
    "source": [
     "## 5. Visualize the model output"
@@ -528,7 +543,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0f74f119",
+   "id": "fc700939",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -562,7 +577,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ecbb47c8",
+   "id": "549d3602",
    "metadata": {},
    "source": [
     "## 6. Export the saved model"
@@ -571,12 +586,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d68692d3",
+   "id": "5a21d129",
    "metadata": {},
    "outputs": [],
    "source": [
     "if not os.path.exists(output_directory):\n",
-    "    !mkdir -p $output_directory\n",
+    "    os.makedirs(output_directory)\n",
     "file_path = \"{}/image_classification.pt\".format(output_directory)\n",
     "torch.save(model.state_dict(), file_path)\n",
     "print(\"Saved to {}\".format(file_path))"
@@ -584,7 +599,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f18599c",
+   "id": "edc0250c",
    "metadata": {},
    "source": [
     "## Dataset citations\n",
@@ -604,14 +619,6 @@
     "url = \"http://download.tensorflow.org/example_images/flower_photos.tgz\" }\n",
     "```"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6c7dd31d",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -630,7 +637,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.5"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/README.md b/docs/notebooks/transfer_learning/pytorch_image_classification/README.md
index cbcc2beeb..11780b0fe 100644
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/README.md
+++ b/docs/notebooks/transfer_learning/pytorch_image_classification/README.md
@@ -1,6 +1,6 @@
 # Transfer Learning for Image Classification using PyTorch
 
-This notebook uses image classification models from torchvision that were originally trained 
+This notebook uses image classification models from Torchvision that were originally trained 
 using ImageNet and does transfer learning with the Food101 dataset, a flowers dataset, or
 a custom image dataset.
 
@@ -13,57 +13,14 @@ The notebook performs the following steps:
 5. Visualize the model output
 6. Export the saved model
 
-
 ## Running the notebook
 
-The instructions below explain how to run the notebook on bare metal using a
-virtual environment. It is assumed that the host has Python3 and wget pre-installed.
-
-1. Get a clone of the Model Zoo repository from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Create a Python3 virtual environment and install the required packages.
+To run the notebook, follow the instructions in `setup.md`.
    
-   You can use virtualenv:
-   ```
-   python3 -m venv intel-pyt-venv
-   source intel-pyt-venv/bin/activate
-   ```
-   Or Anaconda:
-   ```
-   conda create -n intel-pyt python=3.9
-   conda activate intel-pyt
-   ```
-   Then, from inside the activated virtualenv or conda environment run these steps:
-   ```
-   pip install --upgrade pip
-   pip install -r ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_image_classification/requirements.txt
-   ```
-3. Set environment variables for the path to the dataset folder and an output directory.
-   The dataset and output directories can be empty. The notebook will download the
-   dataset to the dataset directory, if it is empty. Subsequent runs will reuse the dataset.
-   ```
-   export DATASET_DIR=<directory to download the dataset>
-   export OUTPUT_DIR=<output directory for the saved model>
-
-   mkdir -p $DATASET_DIR
-   mkdir -p $OUTPUT_DIR
-   ```
-4. Navigate to the notebook directory in your clone of the model zoo repo, and then start the
-   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server)
-   (note: we recommend using the increased `iopub_msg_rate_limit` and `iopub_data_rate_limit` values
-   when using torchvision's model and data download APIs):
-   ```
-   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_image_classification
-   jupyter notebook --NotebookApp.iopub_msg_rate_limit=10000.0 --NotebookApp.iopub_data_rate_limit=1e10 --port 8888
-   ```
-5. Copy and paste the URL from the terminal to your browser to view and run
-   the notebook.
-   
-Dataset Citations:
+## References
 
+Dataset citations:
+```
 @inproceedings{bossard14,
   title = {Food-101 -- Mining Discriminative Components with Random Forests},
   author = {Bossard, Lukas and Guillaumin, Matthieu and Van Gool, Luc},
@@ -77,4 +34,4 @@ title = "Flowers",
 month = "jan",
 year = "2019",
 url = "http://download.tensorflow.org/example_images/flower_photos.tgz" }
-
+```
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py b/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py
index 8823a9d34..290e48aaa 100644
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py
+++ b/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py
@@ -17,11 +17,10 @@
 #
 
 import torch
-import torchvision
 from pydoc import locate
 
 
-# Dictionary of torchvision image classification models
+# Dictionary of Torchvision image classification models
 torchvision_model_map = {
     "resnet18": {
         "classifier": "fc"
@@ -55,11 +54,12 @@
     }
 }
 
+
 def get_retrainable_model(model_name, num_classes, do_fine_tuning=False):
     # Load an image classification model pretrained on ImageNet
     pretrained_model_class = locate('torchvision.models.{}'.format(model_name))
     classifier_layer = torchvision_model_map[model_name]['classifier']
-                                                         
+
     model = pretrained_model_class(pretrained=True)
 
     if not do_fine_tuning:
@@ -74,6 +74,5 @@ def get_retrainable_model(model_name, num_classes, do_fine_tuning=False):
         classifier = getattr(model, classifier_layer)
         num_features = classifier.in_features
         setattr(model, classifier_layer, torch.nn.Linear(num_features, num_classes))
-        
-    return model
 
+    return model
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/requirements.txt b/docs/notebooks/transfer_learning/pytorch_image_classification/requirements.txt
deleted file mode 100644
index 3901133cd..000000000
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/requirements.txt
+++ /dev/null
@@ -1,15 +0,0 @@
-intel-extension-for-pytorch==2.0.100 # fixed for Snyk scans to avoid vulnerability
-ipython==8.10.0 # fixed for Snyk scans to avoid vulnerability
-ipython-genutils==0.2.0
-ipywidgets==7.6.5
-matplotlib==3.5.1
-matplotlib-inline==0.1.3
-notebook==6.5.1
-numpy==1.22.1
-pandas==1.4.2
-Pillow>=9.3.0
-psutil==5.9.0
-torch==2.0.1 # fixed for Snyk scans to avoid vulnerability
-torchvision==0.15.2 # fixed for Snyk scans to avoid vulnerability
-wheel==0.38.0 # Included this to avoid a vulnerability
-urllib3==1.26.8
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/.gitignore b/docs/notebooks/transfer_learning/pytorch_object_detection/.gitignore
new file mode 100644
index 000000000..9f197c95f
--- /dev/null
+++ b/docs/notebooks/transfer_learning/pytorch_object_detection/.gitignore
@@ -0,0 +1 @@
+vision/
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
index 98e2d4748..1b525b7e4 100644
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
@@ -1,13 +1,14 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "dbea021c",
    "metadata": {},
    "source": [
     "# Transfer Learning for Object Detection\n",
     "\n",
-    "This notebook is adapted from a [PyTorch tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) and demonstrates transfer learning with [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) for an object detection task. It uses object detection models from [torchvision](https://pytorch.org/vision/stable/index.html) that were originally trained using [COCO](https://cocodataset.org/) and does transfer learning with the [PennFudan dataset](https://www.cis.upenn.edu/~jshi/ped_html/), available via public download, or the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d), available through torchvision datasets.\n",
+    "This notebook is adapted from a [PyTorch tutorial](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) and demonstrates transfer learning with [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) for an object detection task. It uses object detection models from [Torchvision](https://pytorch.org/vision/stable/index.html) that were originally trained using [COCO](https://cocodataset.org/) and does transfer learning with the [PennFudan dataset](https://www.cis.upenn.edu/~jshi/ped_html/), available via public download, or the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d), available through Torchvision datasets.\n",
     "\n",
     "The notebook performs the following steps:\n",
     "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
@@ -23,7 +24,9 @@
    "id": "360b4222",
    "metadata": {},
    "source": [
-    "## 1. Import dependencies and setup parameters"
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the `setup.md` to setup a PyTorch environment with all the dependencies required to run the notebook."
    ]
   },
   {
@@ -42,6 +45,8 @@
     "from PIL import Image\n",
     "from pydoc import locate\n",
     "import warnings\n",
+    "import wget\n",
+    "from zipfile import ZipFile\n",
     "\n",
     "import torchvision.models.detection as detection\n",
     "from torchvision.utils import make_grid, draw_bounding_boxes\n",
@@ -77,7 +82,10 @@
     "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
     "\n",
     "# Batch size\n",
-    "batch_size = 2"
+    "batch_size = 2\n",
+    "\n",
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
    ]
   },
   {
@@ -105,20 +113,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Get reference scripts from the torchvision repo that are not in the package\n",
+    "# Get reference scripts from the Torchvision repo that are not in the package\n",
     "if not os.path.exists(\"vision\"):\n",
-    "    !git clone --depth 1 --branch v0.11.3 https://github.com/pytorch/vision.git\n",
+    "    !git clone --depth 1 --branch v0.14.1 https://github.com/pytorch/vision.git\n",
     "\n",
     "import sys\n",
     "sys.path.append(\"vision/references/detection\")\n",
     "\n",
     "import utils\n",
-    "import transforms as T\n",
+    "import torchvision.transforms as T\n",
     "\n",
     "# Define transform function for image inputs\n",
     "def get_transform(train):\n",
     "    transforms = []\n",
-    "    transforms.append(T.ToTensor())\n",
+    "    transforms.append(T.PILToTensor())\n",
+    "    transforms.append(T.ConvertImageDtype(torch.float))\n",
     "    if train:\n",
     "        transforms.append(T.RandomHorizontalFlip(0.5))\n",
     "    return T.Compose(transforms)"
@@ -154,9 +163,11 @@
     "\n",
     "dataset_path = os.path.join(dataset_directory, \"PennFudanPed\")\n",
     "if not os.path.exists(dataset_path):\n",
-    "    !wget https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip .\n",
-    "    !unzip PennFudanPed.zip -d $dataset_directory\n",
-    "    !rm PennFudanPed.zip"
+    "    download_url = \"https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip\"\n",
+    "    zip_file = wget.download(download_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
+    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
+    "        zipfile.extractall(path=dataset_directory)"
    ]
   },
   {
@@ -199,12 +210,13 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "44af5c26",
    "metadata": {},
    "source": [
-    "### Option B: Use a torchvision dataset with bounding boxes\n",
-    "Download and use the KITTI dataset from [torchvision datasets](https://pytorch.org/vision/stable/datasets.html). We have provided a custom Kitti() subclass, defined in [dataset_utils.py](dataset_utils.py), that aligns the dataset's labels to the format expected by the pretrained models and reduces the labels to a 3-class problem: Person, Vehicle, and the background class. If the raw data files are not found in the dataset directory they will be downloaded from torchvision. Subsequent runs will reuse the already downloaded dataset."
+    "### Option B: Use a Torchvision dataset with bounding boxes\n",
+    "Download and use the KITTI dataset from [Torchvision datasets](https://pytorch.org/vision/stable/datasets.html). We have provided a custom Kitti() subclass, defined in [dataset_utils.py](dataset_utils.py), that aligns the dataset's labels to the format expected by the pretrained models and reduces the labels to a 3-class problem: Person, Vehicle, and the background class. If the raw data files are not found in the dataset directory they will be downloaded from Torchvision. Subsequent runs will reuse the already downloaded dataset."
    ]
   },
   {
@@ -260,8 +272,10 @@
     "# Get a batch of data\n",
     "images, targets = next(iter(data_loader))\n",
     "images = list(image for image in images)\n",
+    "targets = [{k: v for k, v in t.items()} for t in targets]\n",
     "\n",
     "model.eval()\n",
+    "\n",
     "predictions = model(images)"
    ]
   },
@@ -444,7 +458,7 @@
    "outputs": [],
    "source": [
     "if not os.path.exists(output_directory):\n",
-    "    !mkdir -p $output_directory\n",
+    "    os.makedirs(output_directory)\n",
     "file_path = \"{}/object_detection.pt\".format(output_directory)\n",
     "torch.save(model.state_dict(), file_path)\n",
     "print(\"Saved to {}\".format(file_path))"
@@ -510,7 +524,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.5"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/README.md b/docs/notebooks/transfer_learning/pytorch_object_detection/README.md
index f77b0ee36..d2911dd01 100644
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/README.md
+++ b/docs/notebooks/transfer_learning/pytorch_object_detection/README.md
@@ -1,9 +1,9 @@
 # Transfer Learning for Object Detection using PyTorch
 
-This notebook uses object detection models from torchvision that were originally trained 
-using COCO and does transfer learning with the [PennFudan dataset](https://www.cis.upenn.edu/~jshi/ped_html/), 
-available via public download, or the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d), 
-available through torchvision datasets.
+This notebook uses object detection models from torchvision that were originally trained
+using COCO and does transfer learning with the [PennFudan dataset](https://www.cis.upenn.edu/~jshi/ped_html/),
+available via public download, or the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d),
+available through Torchvision datasets.
 
 The notebook performs the following steps:
 
@@ -17,51 +17,11 @@ The notebook performs the following steps:
 
 ## Running the notebook
 
-The instructions below explain how to run the notebook on bare metal using a
-virtual environment.
+To run the notebook, follow the instructions in `setup.md`.
 
-1. Get a clone of the Model Zoo repository from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Create a Python3 virtual environment and install required packages.
+## References
 
-   You can use virtualenv:
-   ```
-   python3 -m venv intel-pyt-venv
-   source intel-pyt-venv/bin/activate
-   ```
-   Or Anaconda:
-   ```
-   conda create -n intel-pyt python=3.9
-   conda activate intel-pyt
-   ```
-   Then, from inside the activated virtualenv or conda environment run these steps: 
-   ```
-   pip install --upgrade pip
-   pip install -r ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_object_detection/requirements.txt
-   ```
-3. Set environment variables for the path to the dataset folder and an output directory.
-   The dataset and output directories can be empty. The notebook will download the PennFudan 
-   dataset to the dataset directory, if it is empty. Subsequent runs will reuse the dataset.
-   ```
-   export DATASET_DIR=<directory to download the dataset>
-   export OUTPUT_DIR=<output directory for the saved model>
-
-   mkdir -p $DATASET_DIR
-   mkdir -p $OUTPUT_DIR
-   ```
-4. Navigate to the notebook directory in your clone of the model zoo repo, and then start the
-   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server):
-   ```
-   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_object_detection
-   jupyter notebook --port 8888
-   ```
-5. Copy and paste the URL from the terminal to your browser to view and run
-   the notebook.
-   
-Dataset Citations:
+Dataset citations:
 ```
 @InProceedings{10.1007/978-3-540-76386-4_17,
     author="Wang, Liming
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py b/docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py
index 3bf233100..fb3001baa 100644
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py
+++ b/docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py
@@ -18,11 +18,11 @@
 
 import os
 import csv
-import glob
 import numpy as np
 import torch
 import torchvision
 from PIL import Image
+from typing import Any, Tuple
 
 
 COCO_LABELS = [
@@ -44,6 +44,7 @@
     '__background__', 'Person', 'Vehicle'
 ]
 
+
 class PennFudanDataset(torch.utils.data.Dataset):
     def __init__(self, root, transforms):
         self.root = root
@@ -100,13 +101,15 @@ def __getitem__(self, idx):
         target["iscrowd"] = iscrowd
 
         if self.transforms is not None:
-            img, target = self.transforms(img, target)
+            img = self.transforms(img)
+            # target needs augmentations
 
         return img, target
 
     def __len__(self):
         return len(self.imgs)
 
+
 class Kitti(torchvision.datasets.Kitti):
     def _parse_target(self, index: int):
         labels = []
@@ -116,12 +119,12 @@ def _parse_target(self, index: int):
             for line in content:
                 if line[0] in ['Pedestrian', 'Person_sitting', 'Cyclist']:
                     boxes.append([float(x) for x in line[4:8]])
-                    labels.append(1) # Re-label Pedestrian, Person_sitting, Cyclist to Person=1
+                    labels.append(1)  # Re-label Pedestrian, Person_sitting, Cyclist to Person=1
                 if line[0] in ['Car', 'Truck', 'Van', 'Tram']:
                     boxes.append([float(x) for x in line[4:8]])
-                    labels.append(2) # Re-label Car, Truck, Van, Tram to Vehicle=2
-        
-        boxes = torch.FloatTensor(boxes)  
+                    labels.append(2)  # Re-label Car, Truck, Van, Tram to Vehicle=2
+
+        boxes = torch.FloatTensor(boxes)
         target = {}
         target['image_id'] = torch.tensor([index])
         target['boxes'] = boxes
@@ -130,3 +133,28 @@ def _parse_target(self, index: int):
         target['iscrowd'] = torch.zeros((len(target['labels']),), dtype=torch.int64)
 
         return target
+
+    def __getitem__(self, index: int) -> Tuple[Any, Any]:
+        """Get item at a given index.
+
+        Args:
+            index (int): Index
+        Returns:
+            tuple: (image, target), where
+            target is a list of dictionaries with the following keys:
+
+            - type: str
+            - truncated: float
+            - occluded: int
+            - alpha: float
+            - bbox: float[4]
+            - dimensions: float[3]
+            - locations: float[3]
+            - rotation_y: float
+
+        """
+        image = Image.open(self.images[index])
+        target = self._parse_target(index) if self.train else None
+        if self.transforms:
+            image = self.transforms(image)
+        return image, target
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py b/docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py
index d5ed1de37..770248d84 100644
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py
+++ b/docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py
@@ -18,11 +18,9 @@
 
 import math
 import torch
-import torchvision
-import torchvision.models.detection as detection
 
 
-# Dictionary of torchvision object detection models
+# Dictionary of Torchvision object detection models
 torchvision_model_map = {
     "fasterrcnn_resnet50_fpn": {
         "predictor_model": "faster_rcnn.FastRCNNPredictor"
@@ -38,23 +36,23 @@
     }
 }
 
+
 def get_retrainable_model(model_name, num_classes, pretrained_model_class, predictor_class):
     # Load an object detection model pre-trained on COCO
     model = pretrained_model_class(pretrained=True)
-    
+
     if 'fasterrcnn' in model_name:
         in_features = model.roi_heads.box_predictor.cls_score.in_features
         model.roi_heads.box_predictor = predictor_class(in_features, num_classes)
-        
+
     elif 'retinanet' in model_name:
         # To keep the pretrained weights, do not overwrite the classification_head
         in_features = model.head.classification_head.conv[0].in_channels
         num_anchors = model.head.classification_head.num_anchors
         model.head.classification_head.num_classes = num_classes
-        cls_logits = torch.nn.Conv2d(in_features, num_anchors * num_classes, kernel_size = 3, stride=1, padding=1)
+        cls_logits = torch.nn.Conv2d(in_features, num_anchors * num_classes, kernel_size=3, stride=1, padding=1)
         torch.nn.init.normal_(cls_logits.weight, std=0.01)
         torch.nn.init.constant_(cls_logits.bias, -math.log((1 - 0.01) / 0.01))
         model.head.classification_head.cls_logits = cls_logits
-        
-    return model
 
+    return model
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/requirements.txt b/docs/notebooks/transfer_learning/pytorch_object_detection/requirements.txt
deleted file mode 100644
index 0d9ecad79..000000000
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/requirements.txt
+++ /dev/null
@@ -1,16 +0,0 @@
-intel_extension_for_pytorch==1.13.0 # fixed for Snyk scans to avoid vulnerability
-ipywidgets==8.0.2
-matplotlib==3.6.1
-matplotlib-inline==0.1.6
-notebook==6.5.1
-numpy==1.23.4
-opencv-python==4.6.0.66
-pandas==1.5.1
-Pillow>=9.3.0
-psutil==5.9.3
-pycocotools==2.0.5
-tensorflow-addons==0.18.0
-torchvision==0.14.1 # fixed for Snyk scans to avoid vulnerability
-wheel==0.38.0 # Included this to avoid a vulnerability
-tqdm==4.64.1
-urllib3==1.26.12
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb b/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
index 91fbee862..1c6fc7cf1 100644
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
+++ b/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "b65f0c82",
    "metadata": {},
@@ -9,23 +10,25 @@
     "\n",
     "This notebook demonstrates fine tuning pretrained models from [Hugging Face](https://huggingface.co) using text classification datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or a custom dataset. The notebook uses [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch), which extends PyTorch with optimizations for an extra performance boost on Intel hardware.\n",
     "\n",
-    "Please install the dependencies from the [requirements.txt](requirements.txt) file before executing this notebook.\n",
+    "Please install the dependencies from the [setup.md](../../setup.md) file before executing this notebook.\n",
     "\n",
     "The notebook performs the following steps:\n",
     "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
     "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
-    "3. [Get the model and setup the Trainer](#3.-Get-the-model-and-setup-the-Trainer)\n",
-    "4. [Fine tuning and evaluation](#4.-Fine-tuning-and-evaluation)\n",
-    "5. [Export the model](#5.-Export-the-model)\n",
-    "6. [Reload the model and make predictions](#6.-Reload-the-model-and-make-predictions)"
+    "3. [Prepare the model for fine tuning and evaluation](#3.-Prepare-the-model-for-fine-tuning-and-evaluation)\n",
+    "4. [Export the model](#4.-Export-the-model)\n",
+    "5. [Reload the model and make predictions](#5.-Reload-the-model-and-make-predictions)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "454a6685",
    "metadata": {},
    "source": [
-    "## 1. Import dependencies and setup parameters"
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the [setup.md](../../setup.md) to setup a PyTorch environment with all the dependencies required to run the notebook."
    ]
   },
   {
@@ -43,7 +46,12 @@
     "import sys\n",
     "import torch\n",
     "import warnings\n",
+    "import wget\n",
+    "import typing\n",
     "\n",
+    "from tqdm.auto import tqdm\n",
+    "from torch.optim import AdamW\n",
+    "from torch.utils.data import DataLoader\n",
     "from datasets import ClassLabel, load_dataset, load_metric, Split\n",
     "from datasets import logging as datasets_logging\n",
     "from transformers.utils import logging as transformers_logging\n",
@@ -51,9 +59,10 @@
     "    AutoModelForSequenceClassification,\n",
     "    AutoTokenizer,\n",
     "    Trainer,\n",
-    "    TrainingArguments\n",
+    "    TrainingArguments,\n",
+    "    get_scheduler\n",
     ")\n",
-    "from utils import download_and_extract_zip\n",
+    "from zipfile import ZipFile\n",
     "\n",
     "# Set the logging stream to stdout\n",
     "for handler in transformers_logging._get_library_root_logger().handlers:\n",
@@ -84,18 +93,19 @@
     "\n",
     "# Define an output directory\n",
     "output_dir = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
-    "    os.path.join(os.environ[\"HOME\"], \"{}-output\".format(model_name))\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\", model_name)\n",
     "\n",
-    "# Define a cache directory for dataset and pretrained model files\n",
-    "cache_dir = os.environ[\"CACHE_DIR\"] if \"CACHE_DIR\" in os.environ else \\\n",
-    "    os.path.join(os.environ[\"HOME\"], \"pytorch-text-classification-cache\")\n",
+    "# Define a dataset directory\n",
+    "dataset_dir = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
     "\n",
     "print(\"Model name:\", model_name)\n",
     "print(\"Output directory:\", output_dir)\n",
-    "print(\"Cache directory:\", cache_dir)"
+    "print(\"Dataset directory:\", dataset_dir)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "2f258d4f",
    "metadata": {},
@@ -119,7 +129,7 @@
    "outputs": [],
    "source": [
     "# Load the tokenizer\n",
-    "tokenizer = AutoTokenizer.from_pretrained(model_name, cache_dir=cache_dir)\n",
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
     "\n",
     "class TextClassificationData():\n",
     "    \"\"\"\n",
@@ -180,10 +190,11 @@
     "\n",
     "        # Display the sample using a dataframe\n",
     "        sample = pd.DataFrame(dataset_sample, columns=columns)\n",
-    "        return sample.style.hide_index()"
+    "        return sample.style.hide()"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "fd8512e1",
    "metadata": {},
@@ -192,6 +203,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "640e5611",
    "metadata": {},
@@ -200,7 +212,7 @@
     "\n",
     "[Hugging Face Datasets](https://huggingface.co/datasets) has a catalog of datasets that can be specified by name. Information about the dataset is available in the catalog (including information on the size of the dataset and the splits).\n",
     "\n",
-    "The next cell gets the [IMDb movie review dataset](https://huggingface.co/datasets/imdb) using the Hugging Face datasets API. If the notebook is executed multiple times, the dataset will be used from the cache directory, to speed up the time that it takes to run.\n",
+    "The next cell gets the [IMDb movie review dataset](https://huggingface.co/datasets/imdb) using the Hugging Face datasets API. If the notebook is executed multiple times, the dataset will be used from the dataset directory, to speed up the time that it takes to run.\n",
     "\n",
     "The IMDb dataset in Hugging Face has 3 splits: `train`, `test`, and `unsupervised`. This notebook will be using data from the `train` split for training and data from the `test` split for evaluation. The data has 2 columns: `text` (string with the movie review) and `label` (integer class label). The code in the next cell is setup to run using the IMDb dataset, so note that if a different dataset is being used, you may need to change the split names and/or the column names."
    ]
@@ -265,7 +277,7 @@
     "sentence2_key = None\n",
     "label_key = \"label\"\n",
     "\n",
-    "dataset = HFDSTextClassificationData(tokenizer, cache_dir, dataset_name, train_dataset_size, eval_dataset_size,\n",
+    "dataset = HFDSTextClassificationData(tokenizer, dataset_dir, dataset_name, train_dataset_size, eval_dataset_size,\n",
     "                                     Split.TRAIN, Split.TEST, sentence1_key, sentence2_key, label_key)\n",
     "\n",
     "# Print a sample of the data\n",
@@ -273,6 +285,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "362625ec",
    "metadata": {},
@@ -281,6 +294,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "28c0ba36",
    "metadata": {},
@@ -289,7 +303,7 @@
     "\n",
     "Instead of using a dataset from the Hugging Face dataset catalog, a custom dataset from your local system or a download can be used.\n",
     "\n",
-    "In this example, we download the [SMS Spam Collection dataset](https://archive-beta.ics.uci.edu/ml/datasets/sms+spam+collection). The zip file has a single tab-separated value file with two columns. The first column is the label (`ham` or `spam`) and the second column is the text of the SMS message:\n",
+    "In this example, we download the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). (Note: Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.) The zip file has a single tab-separated value file with two columns. The first column is the label (`ham` or `spam`) and the second column is the text of the SMS message:\n",
     "```\n",
     "<ham or spam>\t<text>\n",
     "<ham or spam>\t<text>\n",
@@ -372,8 +386,8 @@
     "\n",
     "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
     "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
-    "dataset_url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip\"\n",
-    "dataset_dir = os.path.join(cache_dir, \"smsspamcollection\")\n",
+    "dataset_url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
+    "dataset_dir = os.path.join(dataset_dir, \"sms_spam_collection\")\n",
     "csv_name = \"SMSSpamCollection\"\n",
     "delimiter = \"\\t\"\n",
     "label_names = [\"ham\", \"spam\"]\n",
@@ -384,7 +398,10 @@
     "# If we don't already have the csv file, download and extract the zip file to get it.\n",
     "if not os.path.exists(os.path.join(dataset_dir, csv_name)) and \\\n",
     "                      not os.path.exists(os.path.join(dataset_dir, renamed_csv)):\n",
-    "    download_and_extract_zip(dataset_url, dataset_dir)\n",
+    "    zip_file = wget.download(dataset_url, dataset_dir)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_dir))\n",
+    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
+    "        zipfile.extractall(path=dataset_dir)\n",
     "\n",
     "if not os.path.exists(os.path.join(dataset_dir, renamed_csv)):\n",
     "    os.rename(os.path.join(dataset_dir, csv_name), os.path.join(dataset_dir, renamed_csv))\n",
@@ -408,11 +425,200 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "e3a24bcd",
    "metadata": {},
    "source": [
-    "## 3. Get the model and setup the Trainer\n",
+    "## 3. Prepare the model for fine tuning and evaluation\n",
+    "\n",
+    "The notebook has two options to train the model.\n",
+    "\n",
+    "- Option A: Use the [`Trainer`](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.Trainer) API from Hugging Face.\n",
+    "- Option B: Use the native PyTorch API.\n",
+    "\n",
+    "In both cases, the model ends up being a transformers model and depending on the class constructor arguments, the appropriate API is selected.\n",
+    "\n",
+    "Execute the following cell to declare the base class used for the Text Classification Model setup."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "797485aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class TextClassificationModel():\n",
+    "    \"\"\"\n",
+    "    Class used for model loading, training and evaluation.\n",
+    "    \"\"\"\n",
+    "    def __init__(self, \n",
+    "                 model_name: str, \n",
+    "                 num_labels: int, \n",
+    "                 training_args: TrainingArguments = None, \n",
+    "                 ipex_optimize: bool = True, \n",
+    "                 device: str = \"cpu\"):\n",
+    "        \"\"\"\n",
+    "        Initialize the TextClassificationModel class for a text classification model with\n",
+    "        PyTorch. The class uses the model_name to load the pre-trained PyTorch model from\n",
+    "        Hugging Face. If the training_args are given then the Trainer API is selected for\n",
+    "        training and evaluation of the model otherwise native PyTorch API is selected for\n",
+    "        model training and evaluation\n",
+    "        \n",
+    "        :param model_name: Name of the pre-trained model to load from Hugging Face\n",
+    "        :param num_labels: Number of class labels\n",
+    "        :param training_args: A TrainingArguments object if using the Trainer API to train\n",
+    "                              the model. If None, native PyTorch API is used for training.\n",
+    "        :param ipex_optimize: If True, then the model is optimized to run on intel hardware.\n",
+    "        :param device: Device to run on the PyTorch model.\n",
+    "        \"\"\"\n",
+    "        self.model_name = model_name\n",
+    "        self.num_labels = num_labels\n",
+    "        self.training_args = training_args\n",
+    "        self.device = device\n",
+    "        self.trainer = None\n",
+    "        \n",
+    "        self.train_ds = dataset.train_ds\n",
+    "        self.eval_ds = dataset.eval_ds\n",
+    "        \n",
+    "        # Load the model using the pretrained weights\n",
+    "        self.model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)\n",
+    "       \n",
+    "        # Apply the ipex optimize function to the model\n",
+    "        if ipex_optimize:\n",
+    "            self.model = ipex.optimize(self.model)\n",
+    "            \n",
+    "    def train(self, \n",
+    "              dataset: TextClassificationData,\n",
+    "              optimizers: typing.Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR],\n",
+    "              num_train_epochs: int = 1,\n",
+    "              batch_size: int = 16,\n",
+    "              compute_metrics: typing.Callable = None,\n",
+    "              shuffle_samples: bool = True\n",
+    "             ):\n",
+    "\n",
+    "        # If training_args are given, we use the `Trainer` API to train the model\n",
+    "        if self.training_args:\n",
+    "            self.model.train()\n",
+    "            self.trainer = Trainer(model=self.model,\n",
+    "                                   args=self.training_args,\n",
+    "                                   train_dataset=self.train_ds,\n",
+    "                                   eval_dataset=self.eval_ds,\n",
+    "                                   optimizers=optimizers,\n",
+    "                                   compute_metrics=compute_metrics)\n",
+    "            self.trainer.train()\n",
+    "            \n",
+    "        # If training_args are not given, we use native PyTorch API to train the model\n",
+    "        else:\n",
+    "            \n",
+    "            # Rename the `label` column to `labels` because the model expects the argument to be named `labels`\n",
+    "            self.train_ds = self.train_ds.rename_column(\"label\", \"labels\")\n",
+    "            \n",
+    "            # Set the format of the dataset to return PyTorch tensors instead of lists\n",
+    "            self.train_ds.set_format(\"torch\")\n",
+    "            \n",
+    "            train_dataloader = DataLoader(self.train_ds, shuffle=shuffle_samples, batch_size=batch_size)\n",
+    "            \n",
+    "            # Unpack the `optimizers` parameter to get optimizer and lr_scheduler\n",
+    "            optimizer, lr_scheduler = optimizers[0], optimizers[1]\n",
+    "            \n",
+    "            # Define number of training steps for the training progress bar\n",
+    "            num_training_steps = num_train_epochs * len(train_dataloader)\n",
+    "            progress_bar = tqdm(range(num_training_steps))\n",
+    "            \n",
+    "            # Training loop\n",
+    "            self.model.to(self.device)\n",
+    "            self.model.train()\n",
+    "            for epoch in range(num_train_epochs):\n",
+    "                for batch in train_dataloader:\n",
+    "                    batch = {k: v.to(self.device) for k, v in batch.items()}\n",
+    "                    outputs = self.model(**batch)\n",
+    "                    loss = outputs.loss\n",
+    "                    loss.backward()\n",
+    "\n",
+    "                    optimizer.step()\n",
+    "                    lr_scheduler.step()\n",
+    "                    optimizer.zero_grad()\n",
+    "                    progress_bar.update(1)\n",
+    "    \n",
+    "    def evaluate(self, batch_size=16):\n",
+    "        \n",
+    "        if self.trainer:\n",
+    "            metrics = self.trainer.evaluate()\n",
+    "            for key in metrics.keys():\n",
+    "                print(\"{}: {}\".format(key, metrics[key]))\n",
+    "        else:\n",
+    "            # Rename the `label` column to `labels` because the model expects the argument to be named `labels`\n",
+    "            self.eval_ds = self.eval_ds.rename_column(\"label\", \"labels\")\n",
+    "            \n",
+    "            # Set the format of the dataset to return PyTorch tensors instead of lists\n",
+    "            self.eval_ds.set_format(\"torch\")\n",
+    "            \n",
+    "            eval_dataloader = DataLoader(self.eval_ds, batch_size=batch_size)\n",
+    "            progress_bar = tqdm(range(len(eval_dataloader)))\n",
+    "            \n",
+    "            metric = load_metric(\"accuracy\")\n",
+    "            self.model.eval()\n",
+    "            for batch in eval_dataloader:\n",
+    "                batch = {k: v.to(self.device) for k, v in batch.items()}\n",
+    "                with torch.no_grad():\n",
+    "                    outputs = self.model(**batch)\n",
+    "\n",
+    "                logits = outputs.logits\n",
+    "                predictions = torch.argmax(logits, dim=-1)\n",
+    "                metric.add_batch(predictions=predictions, references=batch[\"labels\"])\n",
+    "                progress_bar.update(1)\n",
+    "\n",
+    "            print(metric.compute())\n",
+    "            \n",
+    "    def predict(self, raw_input_text):\n",
+    "        if isinstance(raw_input_text, str):\n",
+    "            raw_input_text = list(raw_input_text)\n",
+    "        \n",
+    "        # Encode the raw text using the tokenizer\n",
+    "        encoded_input = tokenizer(raw_text_input, padding=True, return_tensors='pt')\n",
+    "        \n",
+    "        # Input the encoded text(s) to the model and get the predicted results\n",
+    "        self.model.eval()\n",
+    "        output = self.model(**encoded_input)\n",
+    "        _, predictions = torch.max(output.logits, dim=1)\n",
+    "        \n",
+    "        # Translate the predictions to class label strings\n",
+    "        prediction_labels = dataset.class_labels.int2str(predictions)\n",
+    "\n",
+    "        # Create a dataframe to display the results\n",
+    "        result_list = [list(x) for x in zip(raw_text_input, prediction_labels)]\n",
+    "        result_df = pd.DataFrame(result_list, columns=[\"Input Text\", \"Predicted Label\"])\n",
+    "        return result_df.style.hide()\n",
+    "    \n",
+    "    def parameters(self):\n",
+    "        return self.model.parameters()\n",
+    "    \n",
+    "    def save(self, output_dir):\n",
+    "        self.model.save_pretrained(output_dir)\n",
+    "        \n",
+    "    @classmethod\n",
+    "    def load(cls, output_dir):\n",
+    "        return cls(output_dir, num_labels=len(dataset.get_label_names()))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3e8f1edd",
+   "metadata": {},
+   "source": [
+    "Now that the `TextClassificationModel` class is defined, either use Option A to use the [`Trainer`](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.Trainer) API from Hugging Face or Option B to use the native PyTorch API."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "1a606f16",
+   "metadata": {},
+   "source": [
+    "### Option A: Use the [`Trainer`](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.Trainer) API from Hugging Face\n",
     "\n",
     "This step gets the pretrained model from [Hugging Face](https://huggingface.co/models) and sets up the\n",
     "[TrainingArguments](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.TrainingArguments) and the\n",
@@ -427,83 +633,151 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "797485aa",
+   "id": "0d70a4f8",
    "metadata": {},
    "outputs": [],
    "source": [
     "num_train_epochs = 2\n",
-    "\n",
+    "batch_size = 16\n",
     "num_labels = len(dataset.get_label_names())\n",
-    "print(\"Number of labels:\", num_labels)\n",
-    "\n",
-    "# Load the model using the pretrained weights\n",
-    "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)\n",
     "\n",
-    "# Apply the ipex optimize function to the model\n",
-    "model = ipex.optimize(model)\n",
-    "\n",
-    "# Set up the training args\n",
+    "# Define a TrainingArguments object for the Trainer API to use.\n",
     "training_args = TrainingArguments(output_dir=output_dir, num_train_epochs=num_train_epochs)\n",
     "\n",
-    "# Compute metrics used for evaluation\n",
-    "metric = load_metric(\"accuracy\")\n",
+    "# Get the model from Hugging Face. Since we are specifying training_args, the model is trained and\n",
+    "# evaluated with the Trainer API.\n",
+    "model = TextClassificationModel(model_name=model_name, num_labels=num_labels, training_args=training_args)\n",
+    "\n",
+    "# Define model training parameters\n",
+    "learning_rate      = 5e-5\n",
+    "optimizer          = AdamW(model.parameters(), lr=learning_rate)\n",
+    "num_training_steps = num_train_epochs * len(dataset.train_ds)\n",
+    "metric             = load_metric(\"accuracy\")\n",
+    "lr_scheduler       = get_scheduler(\n",
+    "                        name=\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps\n",
+    "                     )\n",
     "\n",
+    "# Helper function for the Trainer API to compute metrics\n",
     "def compute_metrics(eval_pred):\n",
     "    logits, labels = eval_pred\n",
     "    predictions = np.argmax(logits, axis=-1)\n",
-    "    return metric.compute(predictions=predictions, references=labels)\n",
-    "\n",
-    "# Define the Trainer\n",
-    "trainer = Trainer(model=model,\n",
-    "                  args=training_args,\n",
-    "                  train_dataset=dataset.train_ds,\n",
-    "                  eval_dataset=dataset.eval_ds,\n",
-    "                  compute_metrics=compute_metrics)"
+    "    return metric.compute(predictions=predictions, references=labels)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
-   "id": "e3929cd1",
+   "id": "5fcabd97",
    "metadata": {},
    "source": [
-    "## 4. Fine tuning and evaluation\n",
-    "\n",
-    "This step uses the [Trainer](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.Trainer)\n",
-    "defined in the previous step to train and evaluate the model."
+    "**Train and evaluate the model with the Trainer API**"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b591e396",
+   "id": "e1256c40",
    "metadata": {},
    "outputs": [],
    "source": [
-    "%%time\n",
+    "model.train(\n",
+    "    dataset, \n",
+    "    optimizers=(optimizer, lr_scheduler), \n",
+    "    num_train_epochs=num_train_epochs, \n",
+    "    batch_size=batch_size,\n",
+    "    compute_metrics=compute_metrics\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4a7afe59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.evaluate()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "cce10679",
+   "metadata": {},
+   "source": [
+    "### Option B: Use the native PyTorch API\n",
+    "\n",
+    "This step gets the pretrained model from [Hugging Face](https://huggingface.co/models) and uses native PyTorch API to train and evaluate the model.\n",
     "\n",
-    "trainer.train()"
+    "> Note that it is expected to see a warning at this step about some weights not being used. This is because\n",
+    "> the pretraining head from the original model is being replaced with a classification head."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "361ae450",
+   "id": "202f6b01",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Evaluate and print metrics\n",
-    "metrics = trainer.evaluate()\n",
+    "num_train_epochs = 2\n",
+    "batch_size = 16\n",
+    "num_labels = len(dataset.get_label_names())\n",
     "\n",
-    "for key in metrics.keys():\n",
-    "    print(\"{}: {}\".format(key, metrics[key]))"
+    "# Get the model from Hugging Face. Since we are not specifying training_args, the model is trained and\n",
+    "# evaluated with the native PyTorch API.\n",
+    "model = TextClassificationModel(model_name=model_name, num_labels=num_labels)\n",
+    "\n",
+    "# Define model training parameters\n",
+    "learning_rate      = 5e-5\n",
+    "optimizer          = AdamW(model.parameters(), lr=learning_rate)\n",
+    "num_training_steps = num_train_epochs * len(dataset.train_ds)\n",
+    "lr_scheduler       = get_scheduler(\n",
+    "                        name=\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps\n",
+    "                     )"
    ]
   },
   {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "f4821e46",
+   "metadata": {},
+   "source": [
+    "**Train and evaluate the model with the native PyTorch API**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eeb2e694",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.train(\n",
+    "    dataset, \n",
+    "    optimizers=(optimizer, lr_scheduler), \n",
+    "    num_train_epochs=num_train_epochs, \n",
+    "    batch_size=batch_size\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7ff3b884",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.evaluate()"
+   ]
+  },
+  {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "b83b873f",
    "metadata": {},
    "source": [
-    "## 5. Export the model"
+    "## 4. Export the model"
    ]
   },
   {
@@ -514,15 +788,16 @@
    "outputs": [],
    "source": [
     "# Save the model to our output directory\n",
-    "trainer.save_model(output_dir)"
+    "model.save(output_dir)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "49449342",
    "metadata": {},
    "source": [
-    "## 6. Reload the model and make predictions\n",
+    "## 5. Reload the model and make predictions\n",
     "\n",
     "The output directory is used to reload the model. In the next cell, we evalute the reloaded model to verify that we are getting the same metrics that we saw after fine tuning."
    ]
@@ -534,20 +809,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "reloaded_model = AutoModelForSequenceClassification.from_pretrained(output_dir)\n",
-    "\n",
-    "# Set the model in evaluation mode\n",
-    "reloaded_model.eval()\n",
-    "\n",
-    "# Use the reloaded model in the trainer\n",
-    "trainer.model = reloaded_model\n",
-    "reloaded_model_metrics = trainer.evaluate()\n",
-    "\n",
-    "for key in metrics.keys():\n",
-    "    print(\"{}: {}\".format(key, reloaded_model_metrics[key]))"
+    "reloaded_model = TextClassificationModel.load(output_dir)\n",
+    "    \n",
+    "reloaded_model.evaluate()"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "c70a5386",
    "metadata": {},
@@ -580,23 +848,12 @@
     "    # Define your own input text when using another dataset\n",
     "    raw_text_input = []\n",
     "\n",
-    "# Encode the raw text using the tokenizer\n",
-    "encoded_input = tokenizer(raw_text_input, padding=True, return_tensors='pt')\n",
     "\n",
-    "# Send the encoded input to the model and get the predicted results\n",
-    "output = model(**encoded_input)\n",
-    "_, predictions = torch.max(output.logits, dim=1)\n",
-    "\n",
-    "# Translate the predictions to class label strings\n",
-    "prediction_labels = dataset.class_labels.int2str(predictions)\n",
-    "\n",
-    "# Create a dataframe to display the results\n",
-    "result_list = [list(x) for x in zip(raw_text_input, prediction_labels)]\n",
-    "result_df = pd.DataFrame(result_list, columns=[\"Input Text\", \"Predicted Label\"])\n",
-    "result_df.style.hide_index()"
+    "model.predict(raw_text_input)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "bee40324",
    "metadata": {},
@@ -622,7 +879,8 @@
     "  year         = {2012},\n",
     "  howpublished = {UCI Machine Learning Repository}\n",
     "}\n",
-    "```"
+    "```\n",
+    "Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it."
    ]
   }
  ],
@@ -642,7 +900,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/README.md b/docs/notebooks/transfer_learning/pytorch_text_classification/README.md
index 4a46ca840..c5064ea4c 100644
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/README.md
+++ b/docs/notebooks/transfer_learning/pytorch_text_classification/README.md
@@ -1,70 +1,29 @@
-# Text Classifier fine tuning using IMDb with PyTorch
+# Text Classifier fine tuning with PyTorch
 
 This notebook demonstrates fine tuning [pretrained models from Hugging Face](https://huggingface.co/models)
 using text classification datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
 a custom dataset. The [IMDb Larget Movie Review dataset](https://ai.stanford.edu/~amaas/data/sentiment/) is used
-from the Hugging Face Datasets catalog, and the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection)
+from the Hugging Face Datasets catalog, and the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection)
 is used as an example of a custom dataset being loaded from a csv file.
 
 The notebook uses
-[Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch) which extends PyTorch
+[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
 with optimizations for extra performance boost on Intel hardware.
 
 The notebook performs the following steps:
 1. Import dependencies and setup parameters
 2. Prepare the dataset
-3. Get the model and setup the Trainer
-4. Fine tuning and evaluation
-5. Export the model
-6. Reload the model and make predictions
+3. Prepare the Model for Fine Tuning and Evaluation
+4. Export the model
+5. Reload the model and make predictions
 
 ## Running the notebook
 
-The instructions below explain how to run the notebook on bare metal using a
-virtual environment.
+To run the notebook, follow the instructions to install the [Transfer Learning Notebook requirements](setup.md).
 
-1. Get a clone of the Model Zoo repository from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Create a Python3.9 virtual environment and install the dependencies using the `requirements.txt` file.
-   
-   You can use virtualenv:
-   ```
-   python3.9 -m venv intel-pyt-nlp-venv
-   source intel-pyt-nlp-venv/bin/activate
-   ```
-   Or Anaconda:
-   ```
-   conda create -n intel-pyt-nlp python=3.9
-   conda activate intel-pyt-nlp
-   ```
-   Then, from inside the activated virtualenv or conda environment run these steps:
-   ```
-   pip install --upgrade pip
-   pip install -r ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_text_classification/requirements.txt
-   ```
-3. Set environment variables for the path to the cache folder (for storing pretrained
-   models and datasets) and an output directory. Both of these directories can be empty. Subsequent
-   runs can reuse the same cache directory.
-   ```
-   export CACHE_DIR=<cache directory for models and datasets>
-   export OUTPUT_DIR=<output directory for the saved model>
+## References
 
-   mkdir -p $CACHE_DIR
-   mkdir -p $OUTPUT_DIR
-   ```
-4. Navigate to the notebook directory in your clone of the model zoo repo, and then start the
-   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server):
-   ```
-   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/pytorch_text_classification
-   jupyter notebook --port 8888
-   ```
-5. Copy and paste the URL from the terminal to your browser to view and run
-   the notebook.
-   
-Dataset Citation:
+Dataset Citations
 ```
 @InProceedings{maas-EtAl:2011:ACL-HLT2011,
   author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
@@ -85,4 +44,5 @@ Dataset Citation:
   howpublished = {UCI Machine Learning Repository}
 }
 ```
+Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.
 
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/requirements.txt b/docs/notebooks/transfer_learning/pytorch_text_classification/requirements.txt
deleted file mode 100644
index e20acdad2..000000000
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/requirements.txt
+++ /dev/null
@@ -1,11 +0,0 @@
-datasets==2.6.1
-intel-extension-for-pytorch==1.13.0 # fixed for Snyk scans to avoid vulnerability
-ipywidgets==7.6.5
-notebook==6.5.1
-pandas==1.5.1
-psutil==5.9.3
-scikit-learn==1.1.2
-torch==1.13.1 # fixed for Snyk scans to avoid vulnerability
-wheel==0.38.0 # Included this to avoid a vulnerability
-transformers==4.30.0 # fixed for Snyk scans to avoid vulnerability
-wget==3.2
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/utils.py b/docs/notebooks/transfer_learning/pytorch_text_classification/utils.py
deleted file mode 100644
index b05c28cfd..000000000
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/utils.py
+++ /dev/null
@@ -1,41 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-import os
-
-from zipfile import ZipFile
-
-
-def download_and_extract_zip(zip_url, destination_dir):
-    """
-    Downloads and extracts a zip file to the specified destination.
-    """
-    if not os.path.exists(destination_dir):
-        os.makedirs(destination_dir)
-
-    zip_filename = os.path.basename(zip_url)
-
-    # Download the zip file
-    if not os.path.exists(os.path.join(destination_dir, zip_filename)):
-        import wget
-        wget.download(zip_url, destination_dir)
-
-    # Extract the zip file
-    dataset_zip_path = os.path.join(destination_dir, zip_filename)
-    with ZipFile(dataset_zip_path, "r") as zipfile:
-        zipfile.extractall(path=destination_dir)
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb b/docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
new file mode 100644
index 000000000..87739953e
--- /dev/null
+++ b/docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
@@ -0,0 +1,1011 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f720dcc9",
+   "metadata": {},
+   "source": [
+    "## Instruction Tuning for Text Generation using PyTorch and Hugging Face LLMs\n",
+    "\n",
+    "This notebook demonstrates how to instruction-tune pretrained large language models (LLMs) from [Hugging Face](https://huggingface.co) using datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or a custom dataset.\n",
+    "\n",
+    "Please install the dependencies from [setup.md](/notebooks/setup.md) before executing this notebook.\n",
+    "\n",
+    "The notebook performs the following steps:\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
+    "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
+    "    1. [Option A: Use a Hugging Face dataset](#Option-A:-Use-a-Hugging-Face-dataset)\n",
+    "    2. [Option B: Use a custom dataset](#Option-B:-Use-a-custom-dataset)\n",
+    "    3. [Map and tokenize the dataset](#Map-and-tokenize-the-dataset)\n",
+    "    \n",
+    "3. [Prepare the model and test domain knowledge](#3.-Prepare-the-model-and-test-domain-knowledge)\n",
+    "4. [Transfer learning](#4.-Transfer-learning)\n",
+    "5. [Retest domain knowledge](#5.-Retest-domain-knowledge)\n",
+    "6. [Quantize the model](#6.-Quantize-the-model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "be32c377",
+   "metadata": {},
+   "source": [
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the [setup.md](/notebooks/setup.md) to setup a PyTorch environment with all the dependencies required to run the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fcdba859",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import math\n",
+    "import os\n",
+    "import sys\n",
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "import urllib\n",
+    "import warnings\n",
+    "\n",
+    "import datasets\n",
+    "from datasets import load_dataset\n",
+    "from datasets import logging as datasets_logging\n",
+    "\n",
+    "from transformers import (\n",
+    "    AutoModelForCausalLM, \n",
+    "    AutoTokenizer, \n",
+    "    DataCollatorForSeq2Seq, \n",
+    "    TrainingArguments,\n",
+    "    GenerationConfig,\n",
+    "    Trainer\n",
+    ")\n",
+    "\n",
+    "datasets_logging.set_verbosity_error()\n",
+    "warnings.filterwarnings('ignore')\n",
+    "os.environ[\"TRANSFORMERS_NO_ADVISORY_WARNINGS\"] = \"1\"\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2115e4d9",
+   "metadata": {},
+   "source": [
+    "There is an additional [PEFT module](https://github.com/huggingface/peft) required to train models with low-rank adaptation (LoRA)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ebb4fc31",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install peft\n",
+    "\n",
+    "from peft import LoraConfig, TaskType, get_peft_model, PeftModelForCausalLM"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "462dac6d",
+   "metadata": {},
+   "source": [
+    "Specify the name of the pretrained model from Hugging Face to use (https://huggingface.co/docs/transformers/tasks/language_modeling)\n",
+    "\n",
+    "Example: \n",
+    "* distilgpt2\n",
+    "* EleutherAI/gpt-j-6b\n",
+    "* bigscience/bloom-560m\n",
+    "* bigscience/bloomz-560m\n",
+    "* bigscience/bloomz-3b"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e56da576",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_name = \"EleutherAI/gpt-j-6b\"\n",
+    "\n",
+    "# Define an output directory\n",
+    "output_dir = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
+    "\n",
+    "# Define a dataset directory\n",
+    "dataset_dir = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
+    "\n",
+    "print(\"Model name:\", model_name)\n",
+    "print(\"Output directory:\", output_dir)\n",
+    "print(\"Dataset directory:\", dataset_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45144538",
+   "metadata": {},
+   "source": [
+    "## 2. Prepare the dataset\n",
+    "\n",
+    "The notebook has two options for getting a dataset:\n",
+    "* Option A: Use a dataset from the [Hugging Face Datasets catalog](https://huggingface.co/datasets)\n",
+    "* Option B: Use a custom dataset (downloaded from another source or from your local system)\n",
+    "\n",
+    "In both cases, we define objects for the train and (optional) validation splits and tokenize them."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9177f852",
+   "metadata": {},
+   "source": [
+    "### Option A: Use a Hugging Face dataset\n",
+    "\n",
+    "[Hugging Face Datasets](https://huggingface.co/datasets) has a catalog of datasets that can be specified by name. Information about the dataset is available in the catalog (including information on the size of the dataset and the splits). For instruction-tuning, choose a dataset with fields for \"task\"/\"context\"/\"output\" or \"instruction\"/\"context\"/\"response\" or similar.\n",
+    "\n",
+    "```\n",
+    "{\n",
+    "    \"instruction\": \"Convert this sentence into a question.\",\n",
+    "    \"context\": \"He read the book.\",\n",
+    "    \"response\": \"Did he read the book?\"\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "For example: \n",
+    "* databricks/databricks-dolly-15k\n",
+    "* togethercomputer/RedPajama-Data-Instruct \n",
+    "\n",
+    "The next cell gets a dataset from the Hugging Face datasets API. If the notebook is executed multiple times, the dataset will be used from the dataset directory, to speed up the time that it takes to run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "71c2f865",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_name =  'databricks/databricks-dolly-15k'\n",
+    "dataset = load_dataset(dataset_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "23bf150b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# If the dataset does not have a validation split, create one\n",
+    "if 'validation' not in dataset.keys():\n",
+    "    dataset[\"validation\"] = load_dataset(dataset_name, split=f\"train[:25%]\")\n",
+    "    dataset[\"train\"] = load_dataset(dataset_name, split=f\"train[25%:]\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "57b457fd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Inspect a random sample\n",
+    "dataset['train'][3]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df7473c6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Adjust this dictionary for the keys used in your dataset\n",
+    "dataset_schema = {\n",
+    "    \"instruction_key\": \"instruction\", \n",
+    "    \"context_key\": \"context\",\n",
+    "    \"response_key\": \"response\"\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43511d14",
+   "metadata": {},
+   "source": [
+    "Skip ahead to [mapping and tokenizing](#Map-and-tokenize-the-dataset) the dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4321d967",
+   "metadata": {},
+   "source": [
+    "### Option B: Use a custom dataset\n",
+    "\n",
+    "Instead of using a dataset from the Hugging Face dataset catalog, a custom JSON file from your local system or a download can be used.\n",
+    "\n",
+    "In this example, we download an instruction text dataset example, where each record of the dataset contains text fields for \"instruction\", \"input\", and \"output\" like the following:\n",
+    "```\n",
+    "{\n",
+    "    \"instruction\": \"Convert this sentence into a question.\",\n",
+    "    \"input\": \"He read the book.\",\n",
+    "    \"output\": \"Did he read the book?\"\n",
+    "}\n",
+    "```\n",
+    "If you are using a custom dataset or downloaded dataset that has similarly formatted json, you can use the same code as below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7f7ae4ca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Choose a URL to download or skip this cell and provide a local path in the next cell\n",
+    "url = \"https://raw.githubusercontent.com/intel/intel-extension-for-transformers/1.0.1/examples/optimization/pytorch/huggingface/language-modeling/chatbot/intel_domain.json\"\n",
+    "\n",
+    "filename = os.path.basename(url)\n",
+    "destination = os.path.join(dataset_dir, filename)\n",
+    "\n",
+    "# If we don't already have the json file, download it\n",
+    "if not os.path.exists(destination):\n",
+    "    response = urllib.request.urlopen(url)\n",
+    "    data = response.read().decode(\"utf-8\")\n",
+    "    with open(destination, \"w\") as file:\n",
+    "        file.write(data)\n",
+    "    print('Downloaded file to {}'.format(destination))\n",
+    "else:\n",
+    "    print('Using existing file found at {}'.format(destination))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "89996638",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Customize these variables if you want to load data from pre-existing local files\n",
+    "train_file = destination\n",
+    "validation_file = None\n",
+    "\n",
+    "data_files = {}\n",
+    "dataset_args = {}\n",
+    "data_files[\"train\"] = train_file\n",
+    "if validation_file is not None:\n",
+    "    data_files[\"validation\"] = validation_file\n",
+    "extension = (\n",
+    "    train_file.split(\".\")[-1]\n",
+    "    if train_file is not None\n",
+    "    else validation_file.split(\".\")[-1]\n",
+    ")\n",
+    "if extension == \"txt\":\n",
+    "    extension = \"text\"\n",
+    "\n",
+    "dataset = load_dataset(extension, data_files=data_files)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "095feadc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if 'validation' not in dataset.keys():\n",
+    "    dataset[\"validation\"] = load_dataset(extension, data_files=data_files, split=f\"train[:25%]\")\n",
+    "    dataset[\"train\"] = load_dataset(extension, data_files=data_files, split=f\"train[25%:]\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "31f2144e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Inspect a random sample\n",
+    "dataset['train'][3]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "38758ab1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Adjust this dictionary for the keys used in your dataset\n",
+    "dataset_schema = {\n",
+    "    \"instruction_key\": \"instruction\", \n",
+    "    \"context_key\": \"input\",\n",
+    "    \"response_key\": \"output\"\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b741459",
+   "metadata": {},
+   "source": [
+    "### Map and tokenize the dataset\n",
+    "\n",
+    "After describing the schema of your dataset, create formatted prompts out of each example for instruction-tuning. Then tokenize the prompts with the model's tokenizer and concatenate them together into longer sequences to speed up fine-tuning."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f4ea94db",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROMPT_DICT = {\n",
+    "    \"prompt_with_context\": (\n",
+    "        \"Below is an instruction that describes a task, paired with an input that provides further context. \"\n",
+    "        \"Write a response that appropriately completes the request.\\n\\n\"\n",
+    "        \"### Instruction:\\n{{{instruction_key}}}\\n\\n### Context:\\n{{{context_key}}}\\n\\n### Response:\\n{{{response_key}}}\".format(\n",
+    "        **dataset_schema)\n",
+    "    ),\n",
+    "    \"prompt_without_context\": (\n",
+    "        \"Below is an instruction that describes a task. \"\n",
+    "        \"Write a response that appropriately completes the request.\\n\\n\"\n",
+    "        \"### Instruction:\\n{{{instruction_key}}}\\n\\n### Response:\\n{{{response_key}}}\".format(**dataset_schema)\n",
+    "    ),\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "704af2fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def create_prompts(examples):\n",
+    "    prompts = []\n",
+    "    for example in examples:\n",
+    "        prompt_template = PROMPT_DICT[\"prompt_without_context\"] \\\n",
+    "                if (dataset_schema['context_key'] not in example.keys() or \n",
+    "                    example[dataset_schema['context_key']] is None) else PROMPT_DICT[\"prompt_with_context\"]\n",
+    "        prompt = prompt_template.format_map(example)\n",
+    "        prompts.append(prompt)\n",
+    "    return prompts"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1b00d78a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for key in dataset:\n",
+    "    prompts = create_prompts(dataset[key])\n",
+    "    columns_to_be_removed = list(dataset[key].features.keys())\n",
+    "    dataset[key] = dataset[key].add_column(\"prompts\", prompts)\n",
+    "    dataset[key] = dataset[key].remove_columns(columns_to_be_removed)\n",
+    "    \n",
+    "dataset['train'][3]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f194bd26",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
+    "tokenizer.pad_token_id = (0) \n",
+    "tokenizer.padding_side = \"left\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c804c1d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_seq_length = 512\n",
+    "\n",
+    "def tokenize(prompt, add_eos_token=True):\n",
+    "    results = tokenizer(\n",
+    "        prompt,\n",
+    "        truncation=True,\n",
+    "        max_length=max_seq_length,\n",
+    "        padding=False,\n",
+    "        return_tensors=None,\n",
+    "    )\n",
+    "    for i in range(len(results[\"input_ids\"])):\n",
+    "        if (\n",
+    "            results[\"input_ids\"][i][-1] != tokenizer.eos_token_id\n",
+    "            and len(results[\"input_ids\"][i]) < max_seq_length\n",
+    "            and add_eos_token\n",
+    "        ):\n",
+    "            results[\"input_ids\"][i].append(tokenizer.eos_token_id)\n",
+    "            results[\"attention_mask\"][i].append(1)\n",
+    "\n",
+    "    results[\"labels\"] = results[\"input_ids\"].copy()\n",
+    "\n",
+    "    return results\n",
+    "\n",
+    "def preprocess_function(examples):\n",
+    "    return tokenize(examples[\"prompts\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cac026c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenized_dataset = dataset.map(preprocess_function, batched=True, load_from_cache_file=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "84da97f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def concatenate_data(dataset, max_seq_length):\n",
+    "    concatenated_dataset = {}\n",
+    "    for column in dataset.features:\n",
+    "        concatenated_data = [item for sample in dataset[column] for item in sample]\n",
+    "        reshaped_data = [concatenated_data[i*max_seq_length:(i+1)*max_seq_length] \\\n",
+    "            for i in range(len(concatenated_data) // max_seq_length)]\n",
+    "        concatenated_dataset[column] = reshaped_data\n",
+    "    return datasets.Dataset.from_dict(concatenated_dataset)\n",
+    "\n",
+    "tokenized_dataset_ = tokenized_dataset[\"train\"].remove_columns(\"prompts\")\n",
+    "tokenized_dataset[\"train\"] = concatenate_data(tokenized_dataset_, max_seq_length)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "811fab78",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_dataset = tokenized_dataset[\"train\"]\n",
+    "validation_dataset = tokenized_dataset[\"validation\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "577a2709",
+   "metadata": {},
+   "source": [
+    "## 3. Prepare the model and test domain knowledge\n",
+    "\n",
+    "This notebook uses the Hugging Face Trainer API to download a model for Causal Language Modeling and its associated tokenizer. Get the model and look at some output for a sample prompt."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e4f42425",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "resume_from_checkpoint = False  # User adjust as needed\n",
+    "experiment_identifier = 'bf16'  # User adjust as needed\n",
+    "\n",
+    "model_output_dir = os.path.join(output_dir, model_name, experiment_identifier)\n",
+    "print(\"Model will be saved to:\", model_output_dir)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d7089731",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if resume_from_checkpoint:\n",
+    "    try:\n",
+    "        model = AutoModelForCausalLM.from_pretrained(model_output_dir)\n",
+    "    except OSError:\n",
+    "        model = AutoModelForCausalLM.from_pretrained(model_name)\n",
+    "        model = PeftModelForCausalLM.from_pretrained(model, model_output_dir)\n",
+    "else:\n",
+    "    model = AutoModelForCausalLM.from_pretrained(model_name)\n",
+    "\n",
+    "print('Check the model class: {}'.format(type(model)))\n",
+    "print('Check the model data type: {}'.format(model.dtype))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45bf6823",
+   "metadata": {},
+   "source": [
+    "Use this sample prompt or write your own. Tokenize it, send it to the model for text generation, and then decode and print the response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0c2288a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# For intel-domain custom dataset\n",
+    "prompt_template = PROMPT_DICT[\"prompt_with_context\"]\n",
+    "test_example = {dataset_schema['instruction_key']: 'Write a short marketing paragraph about the following product.',\n",
+    "               dataset_schema['context_key']: 'Intel Xeon processor',\n",
+    "               dataset_schema['response_key']: ''}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfb9cc5f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_prompt = prompt_template.format_map(test_example)\n",
+    "\n",
+    "encoded_input = tokenizer(test_prompt, padding=True, return_tensors='pt')\n",
+    "num_tokens = len(encoded_input['input_ids'])\n",
+    "encoded_input"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1b1a210e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generation_config = GenerationConfig(\n",
+    "    temperature=1.0,\n",
+    "    top_p=0.75,\n",
+    "    top_k=40,\n",
+    "    repetition_penalty=1.0,\n",
+    "    num_beams=4\n",
+    ")\n",
+    "\n",
+    "max_new_tokens=128\n",
+    "\n",
+    "output = model.generate(input_ids=encoded_input['input_ids'], \n",
+    "                        generation_config=generation_config, \n",
+    "                        max_new_tokens=max_new_tokens)\n",
+    "\n",
+    "test_output = tokenizer.batch_decode(output)\n",
+    "print(test_output[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "389f1925",
+   "metadata": {},
+   "source": [
+    "## 4. Transfer learning\n",
+    "\n",
+    "Set up the LoRA parameters and get the PEFT model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6b33f75d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Randomly mask the tokens\n",
+    "data_collator = DataCollatorForSeq2Seq(tokenizer, pad_to_multiple_of=8, return_tensors=\"pt\", padding=True)\n",
+    "\n",
+    "from llm_utils import hf_model_map\n",
+    "model_info = hf_model_map[model_name]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b4d8fd64",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if model_info['lora_target_modules']:\n",
+    "    lora_rank = 8  # Rank parameter \n",
+    "    lora_alpha = 32  # Alpha parameter\n",
+    "    lora_dropout = 0.05  # Dropout parameter \n",
+    "    lora_target_modules = model_info['lora_target_modules']\n",
+    "\n",
+    "    # PEFT settings\n",
+    "    peft_config = LoraConfig(\n",
+    "        r=lora_rank,\n",
+    "        lora_alpha=lora_alpha,\n",
+    "        lora_dropout=lora_dropout,\n",
+    "        target_modules=lora_target_modules,\n",
+    "        bias=\"none\",\n",
+    "        task_type=TaskType.CAUSAL_LM,\n",
+    "    )\n",
+    "    model = get_peft_model(model, peft_config)\n",
+    "    model.print_trainable_parameters()\n",
+    "else:\n",
+    "    print('The {} model is not compatible with LoRA'.format(model_name))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "80c433db",
+   "metadata": {},
+   "source": [
+    "Set up Hugging Face training arguments. For improved training time on Intel® fourth generation Xeon processors, you can experiment with `bf16=True` and `use_ipex=True`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "94431240",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "epochs = 3\n",
+    "do_eval = False  # Use the validation dataset to evaluate perplexity\n",
+    "bf16 = True  # Train with bfloat16 precision\n",
+    "use_ipex = False  # Use Intel® Optimization for PyTorch (IPEX)\n",
+    "max_train_samples = None  # Option to truncate training samples for faster sanity checking"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "51f15c04",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "training_args = TrainingArguments(\n",
+    "    output_dir=model_output_dir, \n",
+    "    num_train_epochs=epochs,\n",
+    "    per_device_train_batch_size=8,\n",
+    "    per_device_eval_batch_size=8,\n",
+    "    bf16=bf16,\n",
+    "    use_ipex=use_ipex,\n",
+    "    no_cuda=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3a8cb139",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Optional: \n",
+    "if max_train_samples is not None:\n",
+    "    train_dataset = train_dataset.select(range(max_train_samples))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "55a32718",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize the Trainer\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    train_dataset=train_dataset,\n",
+    "    eval_dataset=validation_dataset if do_eval else None,\n",
+    "    tokenizer=tokenizer,\n",
+    "    data_collator=data_collator\n",
+    ")\n",
+    "\n",
+    "# Train the model\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a42dc699",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if do_eval:\n",
+    "    eval_results = trainer.evaluate()\n",
+    "    print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "30ee003d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save the model to the model_output_dir\n",
+    "model.save_pretrained(training_args.output_dir)\n",
+    "tokenizer.save_pretrained(training_args.output_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c27b079f",
+   "metadata": {},
+   "source": [
+    "## 5. Retest domain knowledge\n",
+    "\n",
+    "Inference with the test prompt to see if the fine-tuned model gives a better response. You may want to train for at least 3 epochs to see improvement."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a9a42b43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.eval()\n",
+    "\n",
+    "if model.dtype == torch.bfloat16:\n",
+    "    with torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16):\n",
+    "        output = model.generate(input_ids=torch.tensor(encoded_input['input_ids']), \n",
+    "                                generation_config=generation_config, \n",
+    "                                max_new_tokens=max_new_tokens)\n",
+    "else:\n",
+    "    output = model.generate(input_ids=torch.tensor(encoded_input['input_ids']), \n",
+    "                            generation_config=generation_config, \n",
+    "                            max_new_tokens=max_new_tokens)\n",
+    "    \n",
+    "retest_output = tokenizer.batch_decode(output)\n",
+    "print(retest_output[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bf79ca2e",
+   "metadata": {},
+   "source": [
+    "## 6. Quantize the model\n",
+    "\n",
+    "If your model is not converted to bfloat16 precision and optimized by IPEX, quantizing it with Intel® Neural Compressor can result in faster inference times with little or no loss in accuracy. The purpose of this experimental section is to help you explore quantization recipes such as [SmoothQuant](https://github.com/intel/neural-compressor/blob/v2.1.1/docs/source/smooth_quant.md) for large language models. As Intel Neural Compressor adds more support for peft models with low-rank adaptation, we will expand this section.\n",
+    "\n",
+    "Import a data loader class that has a format compatible with Intel Neural Compressor."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12ca5f9b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llm_utils import INCDataloader\n",
+    "\n",
+    "calib_dataloader = INCDataloader(train_dataset, tokenizer, batch_size=8, max_seq_length=max_seq_length,\n",
+    "                                 for_calib=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "93fc58e2",
+   "metadata": {},
+   "source": [
+    "Next use the Intel Neural Compressor to benchmark the full precision model to see how it performs, as a baseline.\n",
+    "\n",
+    "> Note that there is a known issue when running Intel Neural Compressor from a notebook that you may sometimes see the error \n",
+    "> `zmq.error.ZMQError: Address already in use`. If you see this error, rerun the cell again."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bcd57798",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from neural_compressor.config import BenchmarkConfig\n",
+    "from neural_compressor.benchmark import fit\n",
+    "\n",
+    "backend = 'ipex' if use_ipex else 'default'\n",
+    "config = BenchmarkConfig(backend=backend, warmup=50, iteration=500)\n",
+    "\n",
+    "results = fit(model=model, config=config, b_dataloader=calib_dataloader)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "201d07a2",
+   "metadata": {},
+   "source": [
+    "Use the Intel Neural Compressor to quantize the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2f915701",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "kl = False # Whether to use kl divergence for calibration\n",
+    "fallback_add = True # Whether to add fp32 fallback option, recommended for gpt-j"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f33bec92",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from neural_compressor import quantization\n",
+    "from neural_compressor.config import PostTrainingQuantConfig\n",
+    "\n",
+    "recipes = {\n",
+    "    \"smooth_quant\": True,\n",
+    "    \"smooth_quant_args\": {\n",
+    "        \"alpha\": 0.5,\n",
+    "        \"folding\": True,\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "op_type_dict = {}\n",
+    "if kl:\n",
+    "    op_type_dict = {'linear': {'activation': {'algorithm': ['kl']}}}\n",
+    "if fallback_add:\n",
+    "    op_type_dict['add'] = {\"weight\": {\"dtype\": [\"fp32\"]}, \"activation\": {\"dtype\": [\"fp32\"]}}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "60d96549",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = PostTrainingQuantConfig(quant_level=1, backend=backend, excluded_precisions=[\"bf16\"],\n",
+    "                                 recipes=recipes, op_type_dict=op_type_dict)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a451b95a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "quantized_model = quantization.fit(model=model, conf=config, calib_dataloader=calib_dataloader)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0e545a6e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Save the quantized model\n",
+    "inc_output_dir = os.path.join(output_dir, model_name, '{}_quantized'.format(experiment_identifier))\n",
+    "quantized_model.save(inc_output_dir)\n",
+    "print(\"Quantized model directory:\", inc_output_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11f0c4ab",
+   "metadata": {},
+   "source": [
+    "Next we use the Intel Neural Compressor to benchmark the quantized model. These metrics can be compared with\n",
+    "the benchmarking metrics from the full precision model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ecf1bad",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from neural_compressor.utils.pytorch import load\n",
+    "\n",
+    "# Load the quantized model using the INC utils method\n",
+    "model_path = os.path.join(inc_output_dir, 'best_model.pt')\n",
+    "quantized_model = load(model_path, model, dataloader=calib_dataloader)\n",
+    "\n",
+    "# Benchmark the quantized model\n",
+    "config = BenchmarkConfig(backend=backend, warmup=50, iteration=500)\n",
+    "results = fit(model=quantized_model, config=config, b_dataloader=calib_dataloader)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c229270d",
+   "metadata": {},
+   "source": [
+    "Lastly, we can inspect the disk size of the pre- and post-quantization model files:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "be50e1c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('The size of the un-compressed model:')\n",
+    "!du -h {model_output_dir}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3646cba4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('The size of the compressed model:')\n",
+    "!du -h {inc_output_dir}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1b60bdac",
+   "metadata": {},
+   "source": [
+    "## Citations\n",
+    "\n",
+    "<b>[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)</b> - Copyright (2023) Databricks, Inc. This dataset was developed at Databricks (https://www.databricks.com) and its use is subject to the CC BY-SA 3.0 license. Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license: Wikipedia (various pages) - https://www.wikipedia.org/ Copyright © Wikipedia editors and contributors.\n",
+    "\n",
+    "\n",
+    "```\n",
+    "@software{together2023redpajama,\n",
+    "  author = {Together Computer},\n",
+    "  title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},\n",
+    "  month = April,\n",
+    "  year = 2023,\n",
+    "  url = {https://github.com/togethercomputer/RedPajama-Data}\n",
+    "}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10fbedc4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/README.md b/docs/notebooks/transfer_learning/pytorch_text_generation/README.md
new file mode 100644
index 000000000..ef3c038e1
--- /dev/null
+++ b/docs/notebooks/transfer_learning/pytorch_text_generation/README.md
@@ -0,0 +1,39 @@
+# Large language model instruction tuning with PyTorch
+
+This notebook demonstrates instruction tuning [pretrained causal language models from Hugging Face](https://huggingface.co/models)
+using text generation datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
+a custom dataset. The [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset is used
+from the Hugging Face Datasets catalog, and an [Intel domain dataset](https://raw.githubusercontent.com/intel/intel-extension-for-transformers/1.0.1/examples/optimization/pytorch/huggingface/language-modeling/chatbot/intel_domain.json)
+is used as an example of a custom dataset being loaded from a json file.
+
+The notebook includes options for bfloat16 precision training,
+[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
+with optimizations for extra performance boost on Intel hardware, and [SmoothQuant quantization with Intel® Neural Compressor](https://github.com/intel/neural-compressor/blob/v2.1.1/docs/source/smooth_quant.md).
+
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+2. Prepare the dataset
+3. Prepare the model and test domain knowledge
+4. Transfer Learning
+5. Retest domain knowledge
+6. Quantize the model
+
+## Running the notebook
+
+To run the notebook, follow the instructions in `setup.md`.
+
+## References
+
+Dataset Citations
+
+<b>[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)</b> - Copyright (2023) Databricks, Inc. This dataset was developed at Databricks (https://www.databricks.com) and its use is subject to the CC BY-SA 3.0 license. Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license: Wikipedia (various pages) - https://www.wikipedia.org/ Copyright © Wikipedia editors and contributors.
+
+```
+@software{together2023redpajama,
+  author = {Together Computer},
+  title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},
+  month = April,
+  year = 2023,
+  url = {https://github.com/togethercomputer/RedPajama-Data}
+}
+```
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/llm_utils.py b/docs/notebooks/transfer_learning/pytorch_text_generation/llm_utils.py
new file mode 100644
index 000000000..6d7bb8632
--- /dev/null
+++ b/docs/notebooks/transfer_learning/pytorch_text_generation/llm_utils.py
@@ -0,0 +1,101 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import math
+import torch
+from torch.nn.functional import pad
+
+# Dictionary of Hugging Face models
+hf_model_map = {
+    "distilgpt2": {
+        "lora_target_modules": [],
+    },
+    "EleutherAI/gpt-j-6b": {
+        "lora_target_modules": ["q_proj", "v_proj"],
+    },
+    "bigscience/bloom-560m": {
+        "lora_target_modules": ["query_key_value"],
+    },
+    "bigscience/bloomz-3b": {
+        "lora_target_modules": ["query_key_value"],
+    },
+    "bigscience/bloomz-560m": {
+        "lora_target_modules": ["query_key_value"],
+    },
+}
+
+
+class INCDataloader:
+    def __init__(self, dataset, tokenizer, batch_size=1, device='cpu',
+                 max_seq_length=512, for_calib=False):
+        self.dataset = dataset
+        self.tokenizer = tokenizer
+        self.device = device
+        self.batch_size = batch_size
+        self.max_seq_length = max_seq_length
+        self.for_calib = for_calib
+        self.length = math.ceil(len(dataset) / self.batch_size)
+        self.pad_len = 196
+
+        self.dataset.set_format(type='torch', columns=['input_ids'])
+
+    def pad_input(self, input):
+        input_id = input['input_ids'].unsqueeze(0)
+        label = input_id[:, -1].to(self.device)
+        pad_len = self.pad_len - input_id.shape[1]
+        label_index = -2 - pad_len
+        input_id = pad(input_id, (0, pad_len), value=1)
+
+        return (input_id, label, label_index)
+
+    def __iter__(self):
+        input_ids = None
+        labels = None
+        label_indices = None
+        for idx, batch in enumerate(self.dataset):
+            input_id, label, label_index = self.pad_input(batch)
+
+            if input_ids is None:
+                input_ids = input_id
+                labels = label
+                label_indices = [label_index]
+            else:
+                input_ids = torch.cat((input_ids, input_id), 0)
+                labels = torch.cat((labels, label), 0)
+                label_indices.append(label_index)
+
+            if (idx + 1) % self.batch_size == 0:
+                if self.for_calib:
+                    if input_ids.shape[1] > self.max_seq_length:
+                        input_ids = input_ids[:, self.max_seq_length]
+                    yield input_ids
+                else:
+                    yield (input_ids, labels, label_indices)
+                input_ids = None
+                labels = None
+                label_indices = None
+        if (idx + 1) % self.batch_size != 0:
+            if self.for_calib:
+                if input_ids.shape[1] > self.max_seq_length:
+                    input_ids = input_ids[:, self.max_seq_length]
+                yield input_ids
+            else:
+                yield (input_ids, labels, label_indices)
+
+    def __len__(self):
+        return self.length
diff --git a/docs/notebooks/transfer_learning/requirements.txt b/docs/notebooks/transfer_learning/requirements.txt
new file mode 100644
index 000000000..51a8785d3
--- /dev/null
+++ b/docs/notebooks/transfer_learning/requirements.txt
@@ -0,0 +1,32 @@
+Pillow~=10.0.1 # upgraded to fix CVEs
+PyYAML~=6.0
+charset-normalizer~=3.1.0
+datasets~=2.12.0
+gin-config~=0.5.0
+intel-extension-for-pytorch==1.13.100
+intel-tensorflow==2.12.0
+ipython-genutils~=0.2.0
+ipython~=8.12.2; python_version<'3.9'
+ipython~=8.13.2; python_version>='3.9'
+ipywidgets~=8.0.6
+jmespath~=1.0.1
+matplotlib-inline~=0.1.6
+matplotlib~=3.7.1
+notebook~=6.5.4
+numpy~=1.23.5
+opencv-python~=4.7.0.72
+pandas~=2.0.1
+psutil~=5.9.5
+pycocotools~=2.0.6
+scikit-learn~=1.2.2
+scipy~=1.10.1
+sentencepiece~=0.1.99
+tensorflow-addons~=0.20.0
+tensorflow-datasets~=4.9.2
+tensorflow-hub~=0.13.0
+torch==1.13.1
+torchvision==0.14.1
+transformers~=4.30.0
+urllib3~=2.0.2
+wget~=3.2
+evaluate==0.4.0
diff --git a/docs/notebooks/transfer_learning/setup.md b/docs/notebooks/transfer_learning/setup.md
new file mode 100644
index 000000000..7e31f9452
--- /dev/null
+++ b/docs/notebooks/transfer_learning/setup.md
@@ -0,0 +1,51 @@
+# Environment Setup and Running the Notebooks
+
+Use the instructions below to install the dependencies required to run the notebooks.
+
+## Set Up Notebook Environment
+
+1. Get a clone of the Model Zoo repository from GitHub:
+   ```
+   git clone https://github.com/IntelAI/models.git intelai_models
+   export MODEL_ZOO_DIR=$(pwd)/intelai_models
+   ```
+2. Create a Python3 virtual environment and install the required packages.
+   
+   You can use virtualenv:
+   ```
+   python3 -m venv intel-pyt-venv
+   source intel-pyt-venv/bin/activate
+   ```
+   Or Anaconda:
+   ```
+   conda create -n intel-pyt python=3.9
+   conda activate intel-pyt
+   ```
+   Then, from inside the activated virtualenv or conda environment run these steps:
+   ```
+   pip install --upgrade pip
+   pip install -r ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/requirements.txt
+   ```
+3. Set environment variables for the path to the dataset folder and an output directory.
+   The dataset and output directories can be empty. The notebook will download the
+   dataset to the dataset directory, if it is empty. Subsequent runs will reuse the dataset.
+   ```
+   export DATASET_DIR=<directory to download the dataset>
+   export OUTPUT_DIR=<output directory for the saved model>
+
+   mkdir -p $DATASET_DIR
+   mkdir -p $OUTPUT_DIR
+   ```
+4. Navigate to the notebook directory in your clone of the model zoo repo, and then start the
+   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server)
+
+   ```
+   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/
+   jupyter notebook --NotebookApp.iopub_msg_rate_limit=10000.0 --NotebookApp.iopub_data_rate_limit=1e10 --port 8888
+   
+   ```
+
+5. Copy and paste the URL from the terminal to your browser to view and run the notebooks.
+
+Once you have the environment and dependencies set up, see the list of available
+[notebooks](/docs/notebooks/transfer_learning/README.md).
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
index 00fb8eaad..ab5e4edb9 100644
--- a/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
@@ -8,7 +8,7 @@
     "\n",
     "This notebook uses a classifier model that was originally trained using [ImageNet](https://image-net.org) and does transfer learning with either a TF dataset or your own raw images.\n",
     "The notebook performs the following steps:\n",
-    "1. [Install dependencies and setup parameters](#1.-Install-dependencies-and-setup-parameters)\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
     "2. [Prepare the dataset](#2.-Prepare-the-dataset) using either a TF dataset or your own images\n",
     "3. [Predict using the original model](#3.-Predict-using-the-original-model)\n",
     "4. [Transfer learning](#4.-Transfer-Learning)\n",
@@ -20,21 +20,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. Install dependencies and setup parameters"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install ipywidgets==7.6.5 \\\n",
-    "             tensorflow_hub==0.12.0 \\\n",
-    "             tensorflow-datasets==4.4.0 \\\n",
-    "             scipy \\\n",
-    "             'pandas>=1.1.5' \\\n",
-    "             'matplotlib>=3.3.4'"
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the `setup.md` to setup a TensorFlow environment with all the dependencies required to run the notebook."
    ]
   },
   {
@@ -53,6 +41,8 @@
     "import pandas as pd\n",
     "\n",
     "from model_util import tfhub_model_map\n",
+    "import wget\n",
+    "from zipfile import ZipFile\n",
     "\n",
     "print('Supported models:')\n",
     "print('\\n'.join(tfhub_model_map.keys()))\n",
@@ -66,7 +56,10 @@
     "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
     "\n",
     "# Batch size\n",
-    "batch_size = 32"
+    "batch_size = 32\n",
+    "\n",
+    "print(\"\\nDataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
    ]
   },
   {
@@ -104,81 +97,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Option A: Use a TF dataset"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "To use a TF Dataset, load from the tensorflow_datasets library, preprocess the images to convert them to float32, and resize the images. This example uses the [Food-101 dataset using the TensorFlow datasets API](https://www.tensorflow.org/datasets/catalog/food101) dataset, but you can choose from a wide variety of [options](https://www.tensorflow.org/datasets/catalog/overview) (click on the \"Image classification\" section). If the dataset is not found in the dataset directory it is downloaded. Subsequent runs will reuse the already downloaded dataset."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "tags": [
-     "remove_for_custom_dataset"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "# Options include: \"food101\", cats_vs_dogs\", \"horses_or_humans\", \"rock_paper_scissors\", and \"tf_flowers\"\n",
-    "tf_dataset = \"food101\"\n",
-    "\n",
-    "# Load the dataset using the TensorFlow datasets API\n",
-    "[train_ds, test_ds], info = tfds.load(tf_dataset,\n",
-    "                       data_dir=dataset_directory,\n",
-    "                       split=[\"train[:75%]\", \"train[:25%]\"],\n",
-    "                       as_supervised=True,\n",
-    "                       shuffle_files=True,\n",
-    "                       with_info=True)\n",
-    "\n",
-    "# Preprocess the images to convert them to float32 and resize the images to match our model\n",
-    "def preprocess_image(image, label):\n",
-    "    image = tf.image.convert_image_dtype(image, tf.float32)\n",
-    "    image = tf.image.resize_with_pad(image, image_size, image_size)\n",
-    "    return (image, label)\n",
-    "\n",
-    "train_ds = train_ds.map(preprocess_image)\n",
-    "test_ds = test_ds.map(preprocess_image)\n",
-    "\n",
-    "print(\"Dataset directory: \", dataset_directory)\n",
-    "print(\"Training dataset size:\", len(train_ds))\n",
-    "print(\"Validation dataset size:\", len(test_ds))\n",
-    "\n",
-    "# Training data is shuffled for randomness\n",
-    "# https://www.tensorflow.org/datasets/keras_example#build_a_training_pipeline\n",
-    "train_ds = train_ds.cache()\n",
-    "train_ds = train_ds.shuffle(info.splits['train'].num_examples)\n",
-    "train_ds = train_ds.batch(batch_size)\n",
-    "train_ds = train_ds.prefetch(tf.data.AUTOTUNE)\n",
-    "\n",
-    "# Test data does not need to be shuffled, and caching is done after batching\n",
-    "# https://www.tensorflow.org/datasets/keras_example#build_an_evaluation_pipeline\n",
-    "test_ds = test_ds.batch(batch_size)\n",
-    "test_ds = test_ds.cache()\n",
-    "test_ds = test_ds.prefetch(tf.data.AUTOTUNE)\n",
-    "\n",
-    "# Get class names for the dataset\n",
-    "class_names = info.features[\"label\"].names\n",
-    "print(\"Number of classes:\", len(class_names))\n",
-    "print('After processing and batching: ', train_ds.element_spec)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Skip to the next step [3. Predict using the original model](#3.-Predict-using-the-original-model) to continue using the TF Dataset."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Option B: Use your own image dataset"
+    "### Option A: Use your own image dataset"
    ]
   },
   {
@@ -218,10 +137,12 @@
    "source": [
     "# Only run this if you want to use the example flowers dataset\n",
     "if not os.path.exists(dataset_subdir):\n",
-    "    os.mkdir(dataset_subdir)\n",
-    "    !apt-get update && apt-get -qq install curl\n",
-    "    !curl https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz -o flower_photos.tgz\n",
-    "    !tar xvf flower_photos.tgz --directory $dataset_directory"
+    "    os.makedirs(dataset_subdir)\n",
+    "    dataset_url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
+    "    zip_file = wget.download(dataset_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
+    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
+    "        zipfile.extractall(path=dataset_directory)"
    ]
   },
   {
@@ -257,13 +178,83 @@
     "print(\"Number of classes:\", len(class_names))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Skip to the next step [3. Predict using the original model](#3.-Predict-using-the-original-model) to continue using your own image dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Option B: Use a TF dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To use a TF Dataset, specify the name of the dataset to load from the TF Datasets catalog, preprocess the images to convert them to float32, and resize the images. This example uses the [Food-101 dataset using the TensorFlow datasets API](https://www.tensorflow.org/datasets/catalog/food101) dataset, but you can choose from a wide variety of [options](https://www.tensorflow.org/datasets/catalog/overview) (click on the \"Image classification\" section). If the dataset is not found in the dataset directory it is downloaded. Subsequent runs will reuse the already downloaded dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Options include: \"food101\", cats_vs_dogs\", \"horses_or_humans\", \"rock_paper_scissors\", and \"tf_flowers\"\n",
+    "tf_dataset = \"food101\"\n",
+    "\n",
+    "# Load the dataset using the TensorFlow datasets API\n",
+    "[train_ds, test_ds], info = tfds.load(tf_dataset,\n",
+    "                       data_dir=dataset_directory,\n",
+    "                       split=[\"train[:75%]\", \"train[75%:]\"],\n",
+    "                       as_supervised=True,\n",
+    "                       shuffle_files=True,\n",
+    "                       with_info=True)\n",
+    "\n",
+    "# Preprocess the images to convert them to float32 and resize the images to match our model\n",
+    "def preprocess_image(image, label):\n",
+    "    image = tf.image.convert_image_dtype(image, tf.float32)\n",
+    "    image = tf.image.resize_with_pad(image, image_size, image_size)\n",
+    "    return (image, label)\n",
+    "\n",
+    "train_ds = train_ds.map(preprocess_image)\n",
+    "test_ds = test_ds.map(preprocess_image)\n",
+    "\n",
+    "print(\"Dataset directory: \", dataset_directory)\n",
+    "print(\"Training dataset size:\", len(train_ds))\n",
+    "print(\"Validation dataset size:\", len(test_ds))\n",
+    "\n",
+    "# Training data is shuffled for randomness\n",
+    "# https://www.tensorflow.org/datasets/keras_example#build_a_training_pipeline\n",
+    "train_ds = train_ds.cache()\n",
+    "train_ds = train_ds.shuffle(info.splits['train'].num_examples)\n",
+    "train_ds = train_ds.batch(batch_size)\n",
+    "train_ds = train_ds.prefetch(tf.data.AUTOTUNE)\n",
+    "\n",
+    "# Test data does not need to be shuffled, and caching is done after batching\n",
+    "# https://www.tensorflow.org/datasets/keras_example#build_an_evaluation_pipeline\n",
+    "test_ds = test_ds.batch(batch_size)\n",
+    "test_ds = test_ds.cache()\n",
+    "test_ds = test_ds.prefetch(tf.data.AUTOTUNE)\n",
+    "\n",
+    "# Get class names for the dataset\n",
+    "class_names = info.features[\"label\"].names\n",
+    "print(\"Number of classes:\", len(class_names))\n",
+    "print('After processing and batching: ', train_ds.element_spec)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## 3. Predict using the original model\n",
     "\n",
-    "Use the classifier model that was trained using ImageNet to do predictions with the dataset and view the results for a single batch. Table below compares the prediction made by the ImageNet trained model with the actual label from the dataset."
+    "Use the classifier model that was trained using ImageNet to do predictions with the dataset and view the results for a single batch."
    ]
   },
   {
@@ -296,29 +287,22 @@
     "predicted_id = np.argmax(predicted_batch, axis=-1)\n",
     "predicted_label_batch = [imagenet_classes[id] for id in predicted_id]\n",
     "\n",
-    "# Create a results table to list out the ImageNet class prediction vs the actual dataset label\n",
-    "results_table = []\n",
-    "for prediction, actual in zip(predicted_label_batch, actual_label_batch):\n",
-    "    results_table.append([prediction, actual])\n",
-    "\n",
-    "pd.DataFrame(results_table, columns=[\"ImageNet Prediction\", \"Actual Label\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plt.figure(figsize=(10,9))\n",
+    "# Visualize the results\n",
+    "plt.figure(figsize=(16,16))\n",
     "plt.subplots_adjust(hspace=0.5)\n",
-    "for n in range(30):\n",
+    "for n in range(min(batch_size, 30)):\n",
     "    plt.subplot(6,5,n+1)\n",
     "    plt.imshow(image_batch[n])\n",
-    "    plt.title(predicted_label_batch[n].title(), fontsize=9)\n",
+    "    correct_prediction = actual_label_batch[n] == predicted_label_batch[n]\n",
+    "    color = \"darkgreen\" if correct_prediction else \"crimson\"\n",
+    "    title = predicted_label_batch[n].title() if correct_prediction else \"{}\\n({})\".format(predicted_label_batch[n], actual_label_batch[n]) \n",
+    "    plt.title(title, fontsize=14, color=color)\n",
     "    plt.axis('off')\n",
-    "_ = plt.suptitle(\"ImageNet predictions\")\n",
-    "plt.show()"
+    "_ = plt.suptitle(\"ImageNet predictions\", fontsize=16)\n",
+    "plt.show()\n",
+    "\n",
+    "print(\"Correct predictions are shown in green\")\n",
+    "print(\"Incorrect predictions are shown in red with the actual label in parenthesis\")"
    ]
   },
   {
@@ -387,14 +371,25 @@
     "  metrics=['acc'])\n",
     "\n",
     "class CollectBatchStats(tf.keras.callbacks.Callback):\n",
-    "  def __init__(self):\n",
-    "    self.batch_losses = []\n",
-    "    self.batch_acc = []\n",
+    "    def __init__(self):\n",
+    "        self.batch_losses = []\n",
+    "        self.batch_acc = []\n",
+    "\n",
+    "    def on_epoch_begin(self, epoch, logs=None):\n",
+    "        self.batch_losses = []\n",
+    "        self.batch_acc = []\n",
+    "\n",
+    "    def on_train_batch_begin(self, batch, logs=None):\n",
+    "        self.model.reset_metrics()\n",
     "\n",
-    "  def on_train_batch_end(self, batch, logs=None):\n",
-    "    self.batch_losses.append(logs['loss'])\n",
-    "    self.batch_acc.append(logs['acc'])\n",
-    "    self.model.reset_metrics()\n",
+    "    def on_train_batch_end(self, batch, logs=None):\n",
+    "        self.batch_losses.append(logs['loss'])\n",
+    "        self.batch_acc.append(logs['acc'])\n",
+    "\n",
+    "    def on_epoch_end(self, epoch, logs=None):\n",
+    "        # Using the average over all batches is also common instead of just the last batch\n",
+    "        logs['loss'] = self.batch_losses[-1]  # np.mean(self.batch_losses)\n",
+    "        logs['acc'] = self.batch_acc[-1]  # np.mean(self.batch_acc)\n",
     "\n",
     "batch_stats_callback = CollectBatchStats()\n",
     "\n",
@@ -439,17 +434,17 @@
     "predicted_label_batch = [class_names[id] for id in predicted_id]\n",
     "\n",
     "# Display the results\n",
-    "plt.figure(figsize=(10,9))\n",
+    "plt.figure(figsize=(16,16))\n",
     "plt.subplots_adjust(hspace=0.5)\n",
-    "for n in range(30):\n",
+    "for n in range(min(batch_size, 30)):\n",
     "    plt.subplot(6,5,n+1)\n",
     "    plt.imshow(image_batch[n])\n",
     "    correct_prediction = actual_label_batch[n] == predicted_label_batch[n]\n",
     "    color = \"darkgreen\" if correct_prediction else \"crimson\"\n",
     "    title = predicted_label_batch[n].title() if correct_prediction else \"{}\\n({})\".format(predicted_label_batch[n], actual_label_batch[n]) \n",
-    "    plt.title(title, fontsize=9, color=color)\n",
+    "    plt.title(title, fontsize=14, color=color)\n",
     "    plt.axis('off')\n",
-    "_ = plt.suptitle(\"Model predictions\")\n",
+    "_ = plt.suptitle(\"Model predictions\", fontsize=16)\n",
     "plt.show()\n",
     "print(\"Correct predictions are shown in green\")\n",
     "print(\"Incorrect predictions are shown in red with the actual label in parenthesis\")"
@@ -493,13 +488,6 @@
     "url = \"http://download.tensorflow.org/example_images/flower_photos.tgz\" }\n",
     "```"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -518,9 +506,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.5"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 1
+ "nbformat_minor": 4
 }
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/README.md b/docs/notebooks/transfer_learning/tf_image_classification/README.md
index 98ba4e3f8..30eaed604 100644
--- a/docs/notebooks/transfer_learning/tf_image_classification/README.md
+++ b/docs/notebooks/transfer_learning/tf_image_classification/README.md
@@ -3,101 +3,21 @@
 This notebook uses transfer learning with multiple [TF Hub](https://tfhub.dev) image classifiers,
 [TF datasets](https://www.tensorflow.org/datasets/), and custom image datasets.
 
-Steps that the notebook follows:
-* Install dependencies and setup parameters
-* Prepare the dataset using either a TF dataset or your own images
-* Predict using the original model
-* Transfer learning
-* Evaluate the model
-* Export the saved model
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+1. Prepare the dataset
+1. Predict using the original model
+1. Transfer Learning
+1. Evaluate the model
+1. Export the saved model
 
 ## Running the notebook
 
-The instructions below explain how to run the notebook on [bare metal](#bare-metal) using a
-virtual environment or in a [docker container](#using-docker).
+To run the notebook, follow the instructions in `setup.md`.
 
-### Bare Metal
+## References
 
-1. Get a clone of the Model Zoo for Intel Architecture repository from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Create a Python3 virtual environment and install `intel-tensorflow` and `notebook`:
-   ```
-   python3 -m venv intel-tf-venv
-   source intel-tf-venv/bin/activate
-   pip install --upgrade pip
-   pip install intel-tensorflow notebook
-   ```
-3. Set environment variables for the path to the dataset folder and an output directory.
-   The dataset and output directories can be empty. The notebook will download the specified
-   dataset to the dataset directory, if it is empty. Subsequent runs will reuse the dataset.
-   ```
-   export DATASET_DIR=<directory to the TF dataset or custom dataset>
-   export OUTPUT_DIR=<output directory for the saved model>
-
-   mkdir -p $DATASET_DIR
-   mkdir -p $OUTPUT_DIR
-   ```
-4. Navigate to the notebook directory in your clone of the model zoo repo, and then start the
-   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server):
-   ```
-   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/tf_image_classification
-   jupyter notebook --port 8888
-   ```
-5. Copy and paste the URL from the terminal to your browser to view and run
-   the notebook.
-
-### Using Docker
-
-1. Get a clone of the Model Zoo for Intel Architecture repository from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Use the `intel/intel-optimized-tensorflow:jupyter` docker container to
-   run the notebook.
-
-   The snippet below shows how to mount directories in the container for your clone
-   of the model zoo, a dataset directory, and a directory for output (like the saved
-   model). The notebook will either download the specified TF dataset or use a custom
-   dataset in the `DATASET_DIR`.
-   ```
-   DATASET_DIR=<directory to the TF dataset or custom dataset>
-   OUTPUT_DIR=<output directory for the saved model>
-
-   mkdir -p $DATASET_DIR
-   mkdir -p $OUTPUT_DIR
-
-   # The notebook directory in your clone of the model zoo
-   NOTEBOOK_DIR=${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/tf_image_classification
-
-   docker run -d --rm \
-     -p 8888:8888 \
-     --name intel-tensorflow-jupyter \
-     --env http_proxy=$https_proxy \
-     --env https_proxy=$https_proxy \
-     --env DATASET_DIR=${DATASET_DIR} \
-     --env OUTPUT_DIR=${OUTPUT_DIR} \
-     -v ${DATASET_DIR}:${DATASET_DIR} \
-     -v ${OUTPUT_DIR}:${OUTPUT_DIR} \
-     -v ${NOTEBOOK_DIR}:/tf \
-     intel/intel-optimized-tensorflow:jupyter
-   ```
-3. After the container starts, view the logs to get the URL and token for
-   the notebook. Copy the URL with the token to view the notebook into your
-   browser to view your notebook (for example:
-   `http://127.0.0.1:8888/?token=<token>`).
-   ```
-   docker logs intel-tensorflow-jupyter
-   ```
-4. When you are done using the notebook, you can stop the container:
-   ```
-   docker stop intel-tensorflow-jupyter
-   ```
-
-## Dataset citations
+Dataset citations
 ```
 @inproceedings{bossard14,
   title = {Food-101 -- Mining Discriminative Components with Random Forests},
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/model_util.py b/docs/notebooks/transfer_learning/tf_image_classification/model_util.py
index 9ef82a006..50f052b0e 100644
--- a/docs/notebooks/transfer_learning/tf_image_classification/model_util.py
+++ b/docs/notebooks/transfer_learning/tf_image_classification/model_util.py
@@ -69,4 +69,3 @@
         "image_size": 331
     }
 }
-
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/README.md b/docs/notebooks/transfer_learning/tfhub_bert/README.md
deleted file mode 100644
index d8dce5863..000000000
--- a/docs/notebooks/transfer_learning/tfhub_bert/README.md
+++ /dev/null
@@ -1,147 +0,0 @@
-# Fine-tuning using BERT models from TF Hub
-
-These notebooks demonstrate fine tuning using various [BERT](https://arxiv.org/abs/1810.04805) models
-from [TF Hub](https://tfhub.dev) using Intel® Optimization for TensorFlow. There are notebooks for the
-following tasks:
-
-| Notebook | Description |
-|----------|-------------|
-| [BERT_Question_Answering.ipynb](BERT_Question_Answering.ipynb) | Demonstrates BERT fine tuning using scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) and the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook allows for selecting a BERT large or BERT base model from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-| [BERT_Binary_Text_Classification.ipynb](BERT_Binary_Text_Classification.ipynb) | Demonstrates BERT binary text classification fine tuning using the [IMDb movie review dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews) from [TensorFlow Datasets](https://www.tensorflow.org/datasets) or a custom dataset. The notebook allows for selecting a BERT encoder (BERT large, BERT base, or small BERT) to use along with a preprocessor from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-
-## Running the notebooks
-
-The instructions below explain how to run the notebooks on [bare metal](#bare-metal) using a
-virtual environment or in a [docker container](#using-docker). The notebook itself will do additional
-installations for dependencies that it needs, but it will assume that TensorFlow has already
-been installed.
-
-### Bare Metal
-
-1. Get a clone of the Model Zoo for Intel Architecture repository or download the notebooks from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Create a Python3 virtual environment and install `intel-tensorflow` and `notebook`:
-   ```
-   python3 -m venv intel-tf-venv
-   source intel-tf-venv/bin/activate
-   pip install --upgrade pip
-   pip install intel-tensorflow==2.8.0 notebook
-   ```
-3. Set environment variables for the path to the working folder and an output directory. Both folders
-   need to be writable. The working directory is used for downloading dependencies needed to run
-   the model, including datasets, vocab files, and config files.
-   ```
-   export WORKING_DIR=<workspace used for datasets and other dependencies>
-   export OUTPUT_DIR=<output directory for checkpoints and the saved model>
-
-   mkdir -p $WORKING_DIR
-   mkdir -p $OUTPUT_DIR
-   ```
-4. Navigate to the notebook directory in your clone of the model zoo repo (or the directory
-   where you downloaded the notebooks), and then start the
-   [notebook server](https://jupyter.readthedocs.io/en/latest/running.html#starting-the-notebook-server):
-   ```
-   cd ${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/tfhub_bert
-   jupyter notebook --port 8888
-   ```
-5. Copy and paste the URL from the terminal to your browser to view and run
-   the notebook.
-
-### Using Docker
-
-1. Get a clone of the Model Zoo for Intel Architecture repository or download the notebooks from GitHub:
-   ```
-   git clone https://github.com/IntelAI/models.git intelai_models
-   export MODEL_ZOO_DIR=$(pwd)/intelai_models
-   ```
-2. Use the `intel/intel-optimized-tensorflow:jupyter` docker container to
-   run the notebook.
-
-   The snippet below shows how to mount directories in the container for your clone
-   of the model zoo, a working directory (used to download the dataset, the vocab file,
-   the bert config, etc), and a directory for output (like checkpoints and the saved
-   model).
-   ```
-   WORKING_DIR=<workspace used for datasets and other dependencies>
-   OUTPUT_DIR=<output directory for the saved model>
-
-   mkdir -p $WORKING_DIR
-   mkdir -p $OUTPUT_DIR
-
-   # The notebook directory in your clone of the model zoo, or your notebook download directory
-   export NOTEBOOK_DIR=${MODEL_ZOO_DIR}/docs/notebooks/transfer_learning/tfhub_bert
-
-   docker run -d --rm \
-     -p 8888:8888 \
-     --name intel-tensorflow-jupyter \
-     --env http_proxy=${https_proxy} \
-     --env https_proxy=${https_proxy} \
-     --env WORKING_DIR=${WORKING_DIR} \
-     --env OUTPUT_DIR=${OUTPUT_DIR} \
-     -v ${WORKING_DIR}:${WORKING_DIR} \
-     -v ${OUTPUT_DIR}:${OUTPUT_DIR} \
-     -v ${NOTEBOOK_DIR}:/tf \
-    intel/intel-optimized-tensorflow:jupyter
-   ```
-3. After the container starts, view the logs to get the URL and token for
-   the notebook. Copy the URL with the token to view the notebook into your
-   browser to view your notebook (for example:
-   `http://127.0.0.1:8888/?token=<token>`).
-   ```
-   docker logs intel-tensorflow-jupyter
-   ```
-4. When you are done using the notebook, you can stop the container:
-   ```
-   docker stop intel-tensorflow-jupyter
-   ```
-
-### References
-
-Dataset citations:
-```
-@article{2016arXiv160605250R,
-       author = { {Rajpurkar}, Pranav and {Zhang}, Jian and {Lopyrev},
-                 Konstantin and {Liang}, Percy},
-        title = "{SQuAD: 100,000+ Questions for Machine Comprehension of Text}",
-      journal = {arXiv e-prints},
-         year = 2016,
-          eid = {arXiv:1606.05250},
-        pages = {arXiv:1606.05250},
-archivePrefix = {arXiv},
-       eprint = {1606.05250},
-}
-
-@InProceedings{maas-EtAl:2011:ACL-HLT2011,
-  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
-  title     = {Learning Word Vectors for Sentiment Analysis},
-  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
-  month     = {June},
-  year      = {2011},
-  address   = {Portland, Oregon, USA},
-  publisher = {Association for Computational Linguistics},
-  pages     = {142--150},
-  url       = {http://www.aclweb.org/anthology/P11-1015}
-}
-
-@misc{misc_sms_spam_collection_228,
-  author       = {Almeida, Tiago},
-  title        = {{SMS Spam Collection}},
-  year         = {2012},
-  howpublished = {UCI Machine Learning Repository}
-}
-```
-
-TensorFlow Model Garden citation:
-```
-@misc{tensorflowmodelgarden2020,
-  author = {Hongkun Yu and Chen Chen and Xianzhi Du and Yeqing Li and
-            Abdullah Rashwan and Le Hou and Pengchong Jin and Fan Yang and
-            Frederick Liu and Jaeyoun Kim and Jing Li},
-  title = {{TensorFlow Model Garden}},
-  howpublished = {\url{https://github.com/tensorflow/models}},
-  year = {2020}
-}
-```
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/BERT_Question_Answering.ipynb b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb
similarity index 80%
rename from docs/notebooks/transfer_learning/tfhub_bert/BERT_Question_Answering.ipynb
rename to docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb
index 614bc904e..71e767b6e 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/BERT_Question_Answering.ipynb
+++ b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb
@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -9,38 +10,22 @@
     "This notebook demonstrates fine tuning BERT models from TF Hub using the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). Scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) are used for preprocessing the training dataset and fine tuning.\n",
     "\n",
     "The notebook performs the following steps:\n",
-    "1. [Install dependencies and setup parameters](#1.-Install-dependencies-and-setup-parameters)\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
     "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
     "3. [Fine tuning and evaluation](#3.-Fine-tuning-and-evaluation)\n",
     "4. [Export the saved model](#4.-Export-the-saved-model)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. Install dependencies and setup parameters\n",
+    "## 1. Import dependencies and setup parameters\n",
     "\n",
-    "The notebook assumes that you have already followed the README.md instructions that install Intel-optimized TensorFlow or use the Intel-optimized TensorFlow jupyter docker container. Additional installations needed to run the notebook are done in the next cell."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install --upgrade -q pip\n",
-    "!pip install -q gin-config==0.5.0 \\\n",
-    "                sentencepiece==0.1.96 \\\n",
-    "                tensorflow-addons==0.15.0 \\\n",
-    "                tensorflow-datasets==4.5.2 \\\n",
-    "                tensorflow-hub==0.12.0 \\\n",
-    "                'pandas>=1.1.5' \\\n",
-    "                pyyaml==6.0 \\\n",
-    "                wget==3.2\n",
-    "!pip install --no-deps -q tf-models-official==2.7.0\n",
-    "!apt-get -q update && apt-get -q install -y git"
+    "This notebook assumes that you have already followed the instructions in the [README.md](/notebooks/README.md) to setup a TensorFlow environment with all the dependencies required to run the notebook.\n",
+    "\n",
+    "It will run one of the supported [BERT models from TF Hub](https://tfhub.dev/google/collections/bert/1). The table below has a list of the available models and links to their URLs in TF Hub."
    ]
   },
   {
@@ -55,32 +40,15 @@
     "import tensorflow as tf\n",
     "import wget\n",
     "\n",
-    "from bert_qa_utils import create_mini_dataset_file, \\\n",
-    "                          display_predictions, \\\n",
-    "                          get_config_and_vocab_from_zip, \\\n",
-    "                          predict_squad_customized\n",
+    "from bert_qa_utils import get_model_map\n",
+    "from zipfile import ZipFile\n",
     "\n",
-    "from bert_utils import get_model_map"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "This notebook will run one of the supported [BERT models from TF Hub](https://tfhub.dev/google/collections/bert/1). The table below has a list of the available models and links to their URLs in TF Hub."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
     "tfhub_model_map, models_df = get_model_map(\"tfhub_bert_model_map_qa.json\", return_data_frame=True)\n",
     "models_df.style.hide(axis=\"index\")"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -104,24 +72,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Define a working directory where the dataset and TensorFlow models repo will be downloaded\n",
-    "if \"WORKING_DIR\" in os.environ and os.environ[\"WORKING_DIR\"] != \"\":\n",
-    "    working_dir = os.environ[\"WORKING_DIR\"]\n",
-    "else:\n",
-    "    working_dir = input(\"Path to a working directory (to download datasets, vocab files, etc): \")\n",
+    "# Define a directory to download the dataset\n",
+    "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
     "\n",
     "# Define an output directory for the saved model to be exported\n",
-    "if \"OUTPUT_DIR\" in os.environ and os.environ[\"OUTPUT_DIR\"] != \"\":\n",
-    "    output_dir = os.environ[\"OUTPUT_DIR\"]\n",
-    "else:\n",
-    "    output_dir = input(\"Path to an output directory (for checkpoints and the saved model): \")\n",
+    "output_directory = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
     "    \n",
     "# Directory for downloading the BERT config and vocab file\n",
-    "bert_dir = os.path.join(working_dir, model_name)\n",
+    "bert_dir = os.path.join(output_directory, model_name)\n",
     "\n",
     "# Output directory for logs and checkpoints generated during training\n",
-    "if not os.path.isdir(output_dir):\n",
-    "    os.makedirs(output_dir)\n",
+    "if not os.path.isdir(output_directory):\n",
+    "    os.makedirs(output_directory)\n",
     "\n",
     "# Directory to download the bert checkpoint zip so to get the vocab.txt and bert_config.json\n",
     "if not os.path.isdir(bert_dir):\n",
@@ -131,22 +95,10 @@
     "tfhub_bert_encoder = tfhub_model_map[model_name][\"bert_encoder\"]\n",
     "checkpoint_url = tfhub_model_map[model_name][\"checkpoint_zip\"]\n",
     "\n",
-    "# Extract the vocab.txt and bert_config.json from the checkpoint zip file\n",
-    "vocab_txt, bert_config = get_config_and_vocab_from_zip(checkpoint_url, bert_dir)\n",
-    "\n",
-    "if not os.path.exists(vocab_txt):\n",
-    "    ValueError(\"The vocab file could not be found at\", vocab_txt)\n",
-    "    \n",
-    "if not os.path.exists(bert_config):\n",
-    "    ValueError(\"The bert config could not be found at\", bert_config)\n",
-    "\n",
-    "# Recommended in a multi-user environment\n",
-    "os.environ[\"TFHUB_CACHE_DIR\"] = working_dir\n",
-    "\n",
     "print(\"Using TF Hub model:\", model_name)\n",
     "print(\"BERT encoder URL:\", tfhub_bert_encoder)\n",
-    "print(\"Vocab file:\", vocab_txt)\n",
-    "print(\"BERT config:\", bert_config)"
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
    ]
   },
   {
@@ -156,18 +108,36 @@
    "outputs": [],
    "source": [
     "# Path where the https://github.com/tensorflow/models repo will be cloned\n",
-    "tf_models_dir = os.path.join(working_dir, \"tensorflow-models\")\n",
+    "tf_models_dir = os.path.join(output_directory, \"tensorflow-models\")\n",
     "os.environ[\"TF_MODELS_DIR\"] = tf_models_dir\n",
-    "tf_models_branch = \"v2.7.0\"\n",
+    "tf_models_branch = \"v2.12.0\"\n",
     "\n",
     "# Clone the TensorFlow models repo\n",
     "if not os.path.exists(tf_models_dir):\n",
     "    !git clone --depth=1 --branch=$tf_models_branch https://github.com/tensorflow/models.git $tf_models_dir\n",
-    "        \n",
-    "os.environ[\"PYTHONPATH\"] = tf_models_dir"
+    "\n",
+    "# Add the TensorFlow models repo to the PYTHONPATH\n",
+    "os.environ[\"PYTHONPATH\"] = \"{}:{}\".format(os.getenv(\"PYTHONPATH\", \"\"), tf_models_dir)\n",
+    "\n",
+    "from bert_qa_utils import create_mini_dataset_file, \\\n",
+    "                          display_predictions, \\\n",
+    "                          get_config_and_vocab_from_zip \n",
+    "\n",
+    "# Extract the vocab.txt and bert_config.json from the checkpoint zip file\n",
+    "vocab_txt, bert_config = get_config_and_vocab_from_zip(checkpoint_url, bert_dir)\n",
+    "\n",
+    "if not os.path.exists(vocab_txt):\n",
+    "    ValueError(\"The vocab file could not be found at:\", vocab_txt)\n",
+    "\n",
+    "if not os.path.exists(bert_config):\n",
+    "    ValueError(\"The bert config could not be found at:\", bert_config)\n",
+    "\n",
+    "print(\"Vocab file:\", vocab_txt)\n",
+    "print(\"BERT config:\", bert_config)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -240,7 +210,7 @@
     "overwrite = False\n",
     "\n",
     "# Dataset download directory\n",
-    "squad_dir = os.path.join(working_dir, \"squad\")\n",
+    "squad_dir = os.path.join(dataset_directory, \"squad\")\n",
     "\n",
     "squad_dev_dataset = os.path.join(squad_dir, \"dev-{}.json\".format(squad_version))\n",
     "squad_train_dataset = os.path.join(squad_dir, \"train-{}.json\".format(squad_version))\n",
@@ -293,6 +263,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -320,7 +291,7 @@
     "predict_batch_size = 4\n",
     "\n",
     "# Directory for checkpoints\n",
-    "checkpoint_dir = os.path.join(output_dir, \"{}_checkpoints\".format(model_name))\n",
+    "checkpoint_dir = os.path.join(output_directory, \"{}_checkpoints\".format(model_name))\n",
     "\n",
     "if os.path.exists(checkpoint_dir):\n",
     "    if len(os.listdir(checkpoint_dir)) > 0:\n",
@@ -329,7 +300,8 @@
     "else:\n",
     "    os.makedirs(checkpoint_dir)\n",
     "\n",
-    "!python $tf_models_dir/official/nlp/bert/run_squad.py \\\n",
+    "os.environ[\"TFHUB_CACHE_DIR\"] = os.path.join(output_directory, \"tfhub_modules\")\n",
+    "!python $tf_models_dir/official/legacy/bert/run_squad.py \\\n",
     "  --mode=train_and_eval \\\n",
     "  --input_meta_data_path=$squad_metadata_path \\\n",
     "  --train_data_path=$train_mini_tfrecords_path \\\n",
@@ -355,6 +327,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -370,9 +343,9 @@
    "outputs": [],
    "source": [
     "import tensorflow as tf\n",
-    "from official.nlp.bert import bert_models\n",
-    "from official.nlp.bert import configs as bert_configs\n",
-    "from official.nlp.bert import model_saving_utils\n",
+    "from official.legacy.bert import bert_models\n",
+    "from official.legacy.bert import configs as bert_configs\n",
+    "from official.legacy.bert import model_saving_utils\n",
     "\n",
     "tf.keras.mixed_precision.set_global_policy('float32')\n",
     "bert_config_obj = bert_configs.BertConfig.from_json_file(bert_config)\n",
@@ -380,7 +353,7 @@
     "                                         max_seq_length,\n",
     "                                         hub_module_url=tfhub_bert_encoder)\n",
     "\n",
-    "saved_model_dir = os.path.join(output_dir, \"{}_saved_model\".format(model_name))\n",
+    "saved_model_dir = os.path.join(output_directory, \"{}_saved_model\".format(model_name))\n",
     "\n",
     "if not os.path.exists(saved_model_dir):\n",
     "    os.makedirs(saved_model_dir)\n",
@@ -389,6 +362,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -417,18 +391,11 @@
     "}\n",
     "```"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -442,7 +409,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/README.md b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/README.md
new file mode 100644
index 000000000..f8c198609
--- /dev/null
+++ b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/README.md
@@ -0,0 +1,43 @@
+# Question Answering fine tuning using TensorFlow
+
+This notebook demonstrates fine tuning using various [BERT](https://arxiv.org/abs/1810.04805) models
+from [TF Hub](https://tfhub.dev) using Intel® Optimization for TensorFlow with the SQuAD dataset.
+
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+1. Prepare the dataset
+1. Fine tuning and evaluation
+1. Export the saved model
+
+## Running the notebooks
+
+To run the notebook, follow the instructions in `setup.md`.
+
+## References
+
+Dataset citations:
+```
+@article{2016arXiv160605250R,
+       author = { {Rajpurkar}, Pranav and {Zhang}, Jian and {Lopyrev},
+                 Konstantin and {Liang}, Percy},
+        title = "{SQuAD: 100,000+ Questions for Machine Comprehension of Text}",
+      journal = {arXiv e-prints},
+         year = 2016,
+          eid = {arXiv:1606.05250},
+        pages = {arXiv:1606.05250},
+archivePrefix = {arXiv},
+       eprint = {1606.05250},
+}
+```
+
+TensorFlow Model Garden citation:
+```
+@misc{tensorflowmodelgarden2020,
+  author = {Hongkun Yu and Chen Chen and Xianzhi Du and Yeqing Li and
+            Abdullah Rashwan and Le Hou and Pengchong Jin and Fan Yang and
+            Frederick Liu and Jaeyoun Kim and Jing Li},
+  title = {{TensorFlow Model Garden}},
+  howpublished = {\url{https://github.com/tensorflow/models}},
+  year = {2020}
+}
+```
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/bert_qa_utils.py b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/bert_qa_utils.py
similarity index 75%
rename from docs/notebooks/transfer_learning/tfhub_bert/bert_qa_utils.py
rename to docs/notebooks/transfer_learning/tfhub_bert/question_answering/bert_qa_utils.py
index 14a9fdf2c..6e7d2508a 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/bert_qa_utils.py
+++ b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/bert_qa_utils.py
@@ -20,14 +20,12 @@
 import json
 import os
 import pandas as pd
-import tensorflow as tf
+import sys
 import wget
-
-from official.common import distribute_utils
-from official.nlp.bert.run_squad_helper import get_dataset_fn
-
 from zipfile import ZipFile
 
+# sys.path.append(os.environ["TF_MODELS_DIR"])
+
 
 def create_mini_dataset_file(original_file, output_file, num_dataset_items, overwrite=False):
     """
@@ -40,37 +38,37 @@ def create_mini_dataset_file(original_file, output_file, num_dataset_items, over
     """
     if not os.path.exists(output_file) or overwrite:
         import random
-        
+
         with open(original_file) as f:
             original_data = json.load(f)
 
         total_len = len(original_data["data"])
-        
+
         if num_dataset_items > total_len:
             raise ValueError("The number of dataset items ({}) cannot be more than the total "
                              "dataset length ({}).".format(num_dataset_items, total_len))
-        
+
         item_indicies = random.sample(range(0, total_len), num_dataset_items)
         print("Total dataset length:", total_len)
         print("Randomly selected dataset indices:", item_indicies)
-        
+
         articles = []
-        
+
         for data_index in item_indicies:
             article = {}
             article["paragraphs"] = original_data["data"][data_index]["paragraphs"]
             article["title"] = original_data["data"][data_index]["title"]
-            
+
             for p in article["paragraphs"]:
                 for qas in p["qas"]:
                     qas["id"] = str(qas["id"])
-            
+
             articles.append(article)
 
         # Add the article to a dictionary for the mini dataset
         mini_data = {}
         mini_data["data"] = articles
-        
+
         # Add on a version
         mini_data["version"] = original_data["version"] if "version" in original_data.keys() else "1.0"
 
@@ -85,7 +83,7 @@ def create_mini_dataset_file(original_file, output_file, num_dataset_items, over
 
 def display_predictions(predict_data_path, results_file_path, n=10):
     """ Displays n number of predictions along with the actual value """
-    
+
     def get_data_list():
         count = 0
         data_list = []
@@ -115,7 +113,7 @@ def get_data_list():
                                        "Predicted Answer",
                                        "Actual Answer(s)"])
     return predict_df.style.hide(axis="index")
-                    
+
 
 def get_config_and_vocab_from_zip(zip_url, bert_dir):
     """
@@ -131,7 +129,7 @@ def get_config_and_vocab_from_zip(zip_url, bert_dir):
     """
     vocab_txt = os.path.join(bert_dir, "vocab.txt")
     bert_config = os.path.join(bert_dir, "bert_config.json")
-    
+
     if not os.path.exists(vocab_txt) or not os.path.exists(bert_config):
         downloaded_file = wget.download(zip_url, bert_dir)
         with ZipFile(downloaded_file, "r") as checkpoint_zip:
@@ -148,60 +146,46 @@ def get_file_from_zip(file_path):
                             if matches:
                                 os.replace(matches[0], file_path)
                         break
-            
+
             if not os.path.exists(vocab_txt):
                 get_file_from_zip(vocab_txt)
-            
+
             if not os.path.exists(bert_config):
                 get_file_from_zip(bert_config)
 
         os.remove(downloaded_file)
-        
-    return vocab_txt, bert_config
 
+    return vocab_txt, bert_config
 
-# This function was taken from the TensorFlow Model Garden repo and adapted
-# to be a utility function that has a string for the strategy, directly passes
-# in the max_seq_length instead of a metadata object, and removes the need for FLAGS
-# being defined (instead just passes in the predict_batch_size as an arg).
-
-# https://github.com/tensorflow/models/blob/v2.7.0/official/nlp/bert/run_squad_helper.py#L176
-def predict_squad_customized(strategy_str, max_seq_length, predict_batch_size,
-                             predict_tfrecord_path, num_steps, squad_model):
-    
-    strategy = distribute_utils.get_distribution_strategy(distribution_strategy=strategy_str)
-    
-    """Make predictions using a Bert-based squad model."""
-    predict_dataset_fn = get_dataset_fn(
-        predict_tfrecord_path,
-        max_seq_length,
-        predict_batch_size,
-        is_training=False)
-    predict_iterator = iter(
-        strategy.distribute_datasets_from_function(predict_dataset_fn))
-
-    @tf.function
-    def predict_step(iterator):
-        """Predicts on distributed devices."""
-
-        def _replicated_step(inputs):
-            """Replicated prediction calculation."""
-            x, _ = inputs
-            unique_ids = x.pop('unique_ids')
-            start_logits, end_logits = squad_model(x, training=False)
-            return dict(
-                unique_ids=unique_ids,
-                start_logits=start_logits,
-                end_logits=end_logits)
-
-        outputs = strategy.run(_replicated_step, args=(next(iterator),))
-        return tf.nest.map_structure(strategy.experimental_local_results, outputs)
-
-    all_results = []
-    for _ in range(num_steps):
-        predictions = predict_step(predict_iterator)
-        for result in get_raw_results(predictions):
-            all_results.append(result)
-        if len(all_results) % 100 == 0:
-            print('Made predictions for %d records.', len(all_results))
-    return all_results
+def get_model_map(json_path, return_data_frame=False):
+    """
+    Gets the model map from the speified json path and loads it into a python dictionary. If the
+    data frame option is enabled, it will also return the list of models in a pandas data frame
+    with column headers so that it can be used to display in a notebook.
+    """
+    with open(json_path) as json_file:
+        tfhub_model_map = json.load(json_file)
+
+    if return_data_frame:
+        # Generate list of model names and URL links to TF Hub based on the model map
+        model_options = [[i,
+                          tfhub_model_map[i]["num_hidden_layers"],
+                          tfhub_model_map[i]["hidden_size"],
+                          tfhub_model_map[i]["num_attention_heads"],
+                          "<a href=\"{0}\" target=\"_blank\">{0}</a>".format(
+                              tfhub_model_map[i]["bert_encoder"])]
+                         for i in tfhub_model_map.keys()]
+
+        if len(model_options) == 0:
+            print("Warning: No models were found in the json file:", json_path)
+
+        pd.set_option('display.max_colwidth', None)
+        models_df = pd.DataFrame(model_options,
+                                 columns=["Model",
+                                          "Hidden layers",
+                                          "Hidden size",
+                                          "Attention heads",
+                                          "TF Hub BERT encoder URL"])
+        return tfhub_model_map, models_df
+    else:
+        return tfhub_model_map
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/tfhub_bert_model_map_qa.json b/docs/notebooks/transfer_learning/tfhub_bert/question_answering/tfhub_bert_model_map_qa.json
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/tfhub_bert_model_map_qa.json
rename to docs/notebooks/transfer_learning/tfhub_bert/question_answering/tfhub_bert_model_map_qa.json
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/BERT_Binary_Text_Classification.ipynb b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb
similarity index 89%
rename from docs/notebooks/transfer_learning/tfhub_bert/BERT_Binary_Text_Classification.ipynb
rename to docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb
index 7b0d80896..93485bf38 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/BERT_Binary_Text_Classification.ipynb
+++ b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb
@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -9,7 +10,7 @@
     "This notebook demonstrates fine tuning BERT models from [TF Hub](https://tfhub.dev) with binary text classification datasets.\n",
     "\n",
     "The notebook performs the following steps:\n",
-    "1. [Install dependencies and setup parameters](#1.-Install-dependencies-and-setup-parameters)\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
     "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
     "3. [Build the model](#3.-Build-the-model)\n",
     "4. [Fine tuning and evaluation](#4.-Fine-tuning-and-evaluation)\n",
@@ -18,12 +19,13 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. Install dependencies and setup parameters\n",
+    "## 1. Import dependencies and setup parameters\n",
     "\n",
-    "The notebook assumes that you have already followed the README.md instructions that install Intel-optimized TensorFlow or use the Intel-optimized TensorFlow jupyter docker container. Additional installations needed to run the notebook are done in the next cell."
+    "This notebook assumes that you have already followed the instructions in the `setup.md` to setup a TensorFlow environment with all the dependencies required to run the notebook."
    ]
   },
   {
@@ -32,13 +34,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!pip install --upgrade -q pip\n",
-    "!pip install -q ipywidgets==7.6.5 \\\n",
-    "                tensorflow-hub==0.12.0 \\\n",
-    "                tensorflow-datasets==4.5.2 \\\n",
-    "                'pandas>=1.1.5' \\\n",
-    "                wget==3.2\n",
-    "!pip install -q tensorflow-text"
+    "import os\n",
+    "import pandas as pd\n",
+    "import tensorflow as tf\n",
+    "import tensorflow_hub as hub\n",
+    "import tensorflow_datasets as tfds\n",
+    "import wget\n",
+    "\n",
+    "from bert_utils import get_model_map\n",
+    "from zipfile import ZipFile"
    ]
   },
   {
@@ -47,20 +51,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
-    "import pandas as pd\n",
-    "import tensorflow as tf\n",
-    "import tensorflow_hub as hub\n",
-    "import tensorflow_datasets as tfds\n",
-    "\n",
     "# Note that tensorflow_text isn't used directly but the import is required to register ops used by the\n",
     "# BERT text preprocessor\n",
-    "import tensorflow_text\n",
-    "\n",
-    "from bert_utils import get_model_map, download_and_extract_zip"
+    "! pip3 install tensorflow-text==2.12 --no-deps\n",
+    "import tensorflow_text"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -79,6 +77,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -102,34 +101,30 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Define a working directory where the dataset will be downloaded\n",
-    "if \"WORKING_DIR\" in os.environ and os.environ[\"WORKING_DIR\"] != \"\":\n",
-    "    working_dir = os.environ[\"WORKING_DIR\"]\n",
-    "else:\n",
-    "    working_dir = input(\"Path to a working directory (to download datasets): \")\n",
+    "# Define a directory to download the dataset\n",
+    "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
     "\n",
     "# Define an output directory for the saved model to be exported\n",
-    "if \"OUTPUT_DIR\" in os.environ and os.environ[\"OUTPUT_DIR\"] != \"\":\n",
-    "    output_dir = os.environ[\"OUTPUT_DIR\"]\n",
-    "else:\n",
-    "    output_dir = input(\"Path to an output directory (for the saved model): \")\n",
+    "output_directory = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
     "\n",
     "# Output directory for logs and checkpoints generated during training\n",
-    "if not os.path.isdir(output_dir):\n",
-    "    os.makedirs(output_dir)\n",
-    "\n",
-    "# Recommended in a multi-user environment\n",
-    "os.environ[\"TFHUB_CACHE_DIR\"] = working_dir\n",
-    "\n",
+    "if not os.path.isdir(output_directory):\n",
+    "    os.makedirs(output_directory)\n",
+    "    \n",
     "tfhub_preprocess = tfhub_model_map[model_name][\"preprocess\"]\n",
     "tfhub_bert_encoder = tfhub_model_map[model_name][\"bert_encoder\"]\n",
     "\n",
     "print(\"Using TF Hub model:\", model_name)\n",
     "print(\"BERT encoder URL:\", tfhub_bert_encoder)\n",
-    "print(\"Preprocessor URL:\", tfhub_preprocess)"
+    "print(\"Preprocessor URL:\", tfhub_preprocess)\n",
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -178,6 +173,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -185,6 +181,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -235,9 +232,9 @@
     "tfds_name=\"imdb_reviews\"\n",
     "\n",
     "# Location where the dataset will be downloaded\n",
-    "dataset_dir = os.path.join(output_dir, tfds_name)\n",
-    "if not os.path.isdir(dataset_dir):\n",
-    "    os.makedirs(dataset_dir)\n",
+    "dataset_directory = os.path.join(dataset_directory, tfds_name)\n",
+    "if not os.path.isdir(dataset_directory):\n",
+    "    os.makedirs(dataset_directory)\n",
     "\n",
     "# Label map for sentiment analysis\n",
     "label_map = {\n",
@@ -246,7 +243,7 @@
     "}\n",
     "    \n",
     "# Initialize the dataset splits using a dataset from the TensorFlow datasets catalog\n",
-    "dataset = TFDSBinaryTextClassificationData(dataset_dir=dataset_dir,\n",
+    "dataset = TFDSBinaryTextClassificationData(dataset_dir=dataset_directory,\n",
     "                                           tfds_name=tfds_name,\n",
     "                                           train_split=\"train[:50%]\",\n",
     "                                           val_split=\"train[:20%]\",\n",
@@ -256,6 +253,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -263,6 +261,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -270,7 +269,7 @@
     "\n",
     "Instead of using a dataset from TensorFlow datasets, another dataset from your local system or a download can be used. \n",
     "\n",
-    "In this example, we download the [SMS Spam Collection dataset](https://archive-beta.ics.uci.edu/ml/datasets/sms+spam+collection). The zip file has a single tab-separated value file with two columns. The first column is the label (`ham` or `spam`) and the second column is the text of the SMS message:\n",
+    "In this example, we download the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). (Note: Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.) The zip file has a single tab-separated value file with two columns. The first column is the label (`ham` or `spam`) and the second column is the text of the SMS message:\n",
     "```\n",
     "<ham or spam>\t<text>\n",
     "<ham or spam>\t<text>\n",
@@ -359,16 +358,19 @@
     "\n",
     "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
     "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
-    "dataset_url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip\"\n",
-    "dataset_dir = os.path.join(working_dir, \"smsspamcollection\")\n",
+    "dataset_url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
+    "dataset_directory = os.path.join(dataset_directory, \"sms_spam_collection\")\n",
     "csv_name = \"SMSSpamCollection\"\n",
     "delimiter = \"\\t\"\n",
     "header = False  # Set to true if the csv file has a header row\n",
-    "csv_path = os.path.join(dataset_dir, csv_name)\n",
+    "csv_path = os.path.join(dataset_directory, csv_name)\n",
     "\n",
     "# If we don't already have the csv file, download and extract the zip file to get it.\n",
     "if not os.path.exists(csv_path):\n",
-    "    download_and_extract_zip(dataset_url, dataset_dir)\n",
+    "    zip_file = wget.download(dataset_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
+    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
+    "        zipfile.extractall(path=dataset_directory)\n",
     "\n",
     "# Define the label map for your dataset. The label map below is for the SMS Spam Collection dataset.\n",
     "# The labels defined in this dictionary should match the labels in the csv file.\n",
@@ -398,6 +400,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -426,6 +429,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -461,6 +465,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -480,6 +485,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -520,6 +526,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -535,7 +542,7 @@
    "outputs": [],
    "source": [
     "model_dir = \"{}_{}\".format(model_name, dataset.dataset_name)\n",
-    "model_dir = os.path.join(output_dir, model_dir)\n",
+    "model_dir = os.path.join(output_directory, model_dir)\n",
     "classifier_model.save(model_dir, include_optimizer=False)\n",
     "\n",
     "saved_model_path = os.path.join(model_dir, \"saved_model.pb\")\n",
@@ -544,6 +551,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -562,6 +570,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -607,6 +616,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -631,13 +641,15 @@
     "  year         = {2012},\n",
     "  howpublished = {UCI Machine Learning Repository}\n",
     "}\n",
-    "```"
+    "```\n",
+    "\n",
+    "Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it."
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -651,7 +663,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.9.17"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb
new file mode 100644
index 000000000..6e08dfd89
--- /dev/null
+++ b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb
@@ -0,0 +1,659 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Multiclass text classification using BERT models from TF Hub\n",
+    "\n",
+    "This notebook demonstrates fine tuning BERT models from [TF Hub](https://tfhub.dev) with multiclass text classification datasets.\n",
+    "\n",
+    "The notebook performs the following steps:\n",
+    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
+    "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
+    "3. [Build the model](#3.-Build-the-model)\n",
+    "4. [Fine tuning and evaluation](#4.-Fine-tuning-and-evaluation)\n",
+    "5. [Export the model](#5.-Export-the-model)\n",
+    "6. [Reload the model and make predictions](#6.-Reload-the-model-and-make-predictions)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Import dependencies and setup parameters\n",
+    "\n",
+    "This notebook assumes that you have already followed the instructions in the [README.md](/notebooks/README.md) to setup a TensorFlow environment with all the dependencies required to run the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import pandas as pd\n",
+    "import tensorflow as tf\n",
+    "import tensorflow_hub as hub\n",
+    "import tensorflow_datasets as tfds\n",
+    "\n",
+    "from bert_utils import get_model_map"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Note that tensorflow_text isn't used directly but the import is required to register ops used by the\n",
+    "# BERT text preprocessor\n",
+    "! pip3 install tensorflow-text==2.12.0 --no-deps\n",
+    "import tensorflow_text"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook will run one of the supported [BERT models from TF Hub](https://tfhub.dev/google/collections/bert/1). The table below has a list of the available models and links to their URLs in TF Hub."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the TF Hub model map from json and print a list of the supported models\n",
+    "tfhub_model_map, models_df = get_model_map(\"tfhub_bert_model_map_classifier.json\", return_data_frame=True)\n",
+    "models_df.style.hide(axis=\"index\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Specify the name of the BERT model to use. This string must match one of the models listed in the table above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_name = \"small_bert/bert_en_uncased_L-2_H-128_A-2\"\n",
+    "if model_name not in tfhub_model_map.keys():\n",
+    "    raise ValueError(\"The specified model name ({}) is not supported\".format(model_name))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define a directory to download the dataset\n",
+    "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
+    "\n",
+    "# Define an output directory for the saved model to be exported\n",
+    "output_directory = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
+    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
+    "\n",
+    "# Output directory for logs and checkpoints generated during training\n",
+    "if not os.path.isdir(output_directory):\n",
+    "    os.makedirs(output_directory)\n",
+    "    \n",
+    "tfhub_preprocess = tfhub_model_map[model_name][\"preprocess\"]\n",
+    "tfhub_bert_encoder = tfhub_model_map[model_name][\"bert_encoder\"]\n",
+    "\n",
+    "print(\"Using TF Hub model:\", model_name)\n",
+    "print(\"BERT encoder URL:\", tfhub_bert_encoder)\n",
+    "print(\"Preprocessor URL:\", tfhub_preprocess)\n",
+    "print(\"Dataset directory:\", dataset_directory)\n",
+    "print(\"Output directory:\", output_directory)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Prepare the dataset\n",
+    "\n",
+    "This notebook gets the dataset from a text file or from the [TensorFlow Datasets catalog](https://www.tensorflow.org/datasets/catalog/overview).\n",
+    "\n",
+    "The code ends up defining [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) objects for each split (train, validation, and test) and a map for the translating the numerical to string label.\n",
+    "\n",
+    "Execute the following cell to set the batch size and declare the base class used for the dataset setup."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define the dataset batch size\n",
+    "batch_size = 32\n",
+    "\n",
+    "# Base class used for defining the multi text classification dataset being used\n",
+    "class MultiTextClassificationData():\n",
+    "    def __init__(self, batch_size, label_map):\n",
+    "        self.batch_size = batch_size\n",
+    "        self.label_map = label_map\n",
+    "        self.reverse_label_map = {}\n",
+    "        self.train_ds = None\n",
+    "        self.val_ds = None\n",
+    "        self.test_ds = None\n",
+    "        self.dataset_name = \"\"\n",
+    "        \n",
+    "        for k, v in self.label_map.items():\n",
+    "            self.reverse_label_map[v] = k\n",
+    "        \n",
+    "    def get_str_label(self, numerical_value):\n",
+    "        if not isinstance(numerical_value, int):\n",
+    "            numerical_value = int(tf.math.round(numerical_value))\n",
+    "        \n",
+    "        if numerical_value in self.label_map.keys():\n",
+    "            return self.label_map[numerical_value]\n",
+    "        else:\n",
+    "            raise ValueError(\"The key {} was not found in the label map\".format(numerical_value))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Option A: Use a TensorFlow dataset\n",
+    "\n",
+    "[TensorFlow Datasets](https://www.tensorflow.org/datasets) has a [catalog of datasets](https://www.tensorflow.org/datasets/catalog/overview) that can be specified by name. Information about the dataset is available in the catalog (including information on the size of the dataset and the splits).\n",
+    "\n",
+    "The next cell demonstrates using the [`ag_news_subset`](https://www.tensorflow.org/datasets/catalog/ag_news_subset) dataset from the TensorFlow datasets catalog to get splits for training, validation, and test."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class TFDSMultiTextClassificationData(MultiTextClassificationData):\n",
+    "    def __init__(self, dataset_dir, tfds_name, train_split, val_split, test_split, label_map, batch_size):\n",
+    "        \"\"\"\n",
+    "        Intialize the TFDSMultiTextClassificationData class for a dataset multi text classification dataset\n",
+    "        from the TensorFlow dataset catalog.\n",
+    "        \n",
+    "        :param dataset_dir: Path to a dataset directory to read/write data\n",
+    "        :param tfds_name: String name of the TensorFlow dataset to load\n",
+    "        :param train_split: String specifying which split to load for training (e.g. \"train[:80%]\"). See the\n",
+    "                            https://www.tensorflow.org/datasets/splits documentation for more information on\n",
+    "                            defining splits.\n",
+    "        :param val_split: String specifying the split to load for validation.\n",
+    "        :param test_split: String specifying the split to load for test.\n",
+    "        :param label_map: Dictionary where the key is a numerical value and the value is the string label\n",
+    "        :param batch_size: Batch size\n",
+    "        \"\"\"\n",
+    "        # Init base class\n",
+    "        MultiTextClassificationData.__init__(self, batch_size, label_map) \n",
+    "        \n",
+    "        [self.train_ds, self.val_ds, self.test_ds], info = tfds.load(tfds_name,\n",
+    "                     data_dir=dataset_dir,\n",
+    "                     split=[train_split, val_split, test_split],\n",
+    "                     batch_size=batch_size,\n",
+    "                     as_supervised=True,\n",
+    "                     shuffle_files=True,\n",
+    "                     with_info=True)\n",
+    "        self.dataset_name = tfds_name\n",
+    "        print(info)\n",
+    "\n",
+    "\n",
+    "# Name of the TFDS to use\n",
+    "tfds_name=\"ag_news_subset\"\n",
+    "\n",
+    "# Location where the dataset will be downloaded\n",
+    "dataset_directory = os.path.join(dataset_directory, tfds_name)\n",
+    "if not os.path.isdir(dataset_directory):\n",
+    "    os.makedirs(dataset_directory)\n",
+    "\n",
+    "# Label map for sentiment analysis\n",
+    "label_map = {\n",
+    "    0: \"World\",\n",
+    "    1: \"Sports\",\n",
+    "    2: \"Business\",\n",
+    "    3: \"Sci/Tech\"\n",
+    "}\n",
+    "    \n",
+    "# Initialize the dataset splits using a dataset from the TensorFlow datasets catalog\n",
+    "dataset = TFDSMultiTextClassificationData(dataset_dir=dataset_directory,\n",
+    "                                           tfds_name=tfds_name,\n",
+    "                                           train_split=\"train[:50%]\",\n",
+    "                                           val_split=\"train[:20%]\",\n",
+    "                                           test_split=\"test[:20%]\",\n",
+    "                                           label_map=label_map,\n",
+    "                                           batch_size=batch_size)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Skip to the next step [3. Build the model](#3.-Build-the-model) to continue using the TF dataset."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Option B: Use your own dataset\n",
+    "Instead of using a dataset from TensorFlow datasets, another dataset from your local system or a download can be used.\n",
+    "\n",
+    "In this example, we download the Conference Title dataset. This is a single tab-separated value file with two columns. The first column is the conference title and the second column is the label (VLDB, ISCAS, SIGGRAPH, INFOCOM, WWW):\n",
+    "\n",
+    "```\n",
+    "<conference title>\t<label>\n",
+    "<conference title>\t<label>\n",
+    "<conference title>\t<label>\n",
+    "...\n",
+    "```\n",
+    "\n",
+    "If you are using a custom dataset that has a similarly formatted csv or tsv file, you can still use the class defined below. Just create your object passing in custom values for delimiter, header (whether the file has a header row), the label map, mapping function, etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class CustomCsvMultiTextClassificationData(MultiTextClassificationData):\n",
+    "    def __init__(self, csv_file, delimiter, header, train_percent, val_percent,\n",
+    "                 test_percent, label_map, batch_size, dataset_name, map_function=None):\n",
+    "        \"\"\"\n",
+    "        Intialize the CustomCsvMultiTextClassificationData class for a dataset multi text\n",
+    "        classification dataset that uses a single csv file.\n",
+    "        \n",
+    "        :param csv_file: Path to the csv file\n",
+    "        :param delimiter: String character that separates the fields in each row\n",
+    "        :param header: Boolean indicating whether or not the csv file has a header line that should be skipped\n",
+    "        :param train_percent: Decimal value for the percentage of the dataset that should be used for training\n",
+    "                              (e.g. 0.8 for 80%)\n",
+    "        :param val_percent: Decimal value for the percentage of the dataset that should be used for validation\n",
+    "                            (e.g. 0.1 for 10%)\n",
+    "        :param test_percent: Decimal value for the percentage of the dataset that should be used for test\n",
+    "                             (e.g. 0.1 for 10%)\n",
+    "        :param label_map: Dictionary where the key is a numerical value and the value is the string label\n",
+    "        :param batch_size: Batch size\n",
+    "        :param dataset_name: Name of the dataset. This is used later in this notebook for naming the saved model\n",
+    "                             export folder and determining which input strings to use when testing the reloaded model\n",
+    "        :param map_function: (Optional) If the csv file has string labels instead of the numerical values, provide a\n",
+    "                             map function to apply on the dataset\n",
+    "        \"\"\"\n",
+    "        # Init base class\n",
+    "        MultiTextClassificationData.__init__(self, batch_size, label_map)\n",
+    "        \n",
+    "        self.dataset_name = dataset_name\n",
+    "        \n",
+    "        if (train_percent + val_percent + test_percent) > 1:\n",
+    "            raise ValueError(\"The combined value of the train percentage, validation percentage, and \" \\\n",
+    "                             \"test percentage cannot be greater than 1\")\n",
+    "        \n",
+    "        if not os.path.exists(csv_file):\n",
+    "            raise FileNotFoundError(\"Unable to find the csv file at\", csv_file)      \n",
+    "\n",
+    "        custom_dataset = tf.data.experimental.CsvDataset(filenames=csv_file,\n",
+    "                                                         record_defaults=[tf.string, tf.string],\n",
+    "                                                         field_delim=delimiter,\n",
+    "                                                         use_quote_delim=True,\n",
+    "                                                         header=header)\n",
+    "\n",
+    "        # Count the number of lines in the csv file to get the dataset length\n",
+    "        custom_dataset_len = sum(1 for line in open(csv_file))\n",
+    "\n",
+    "        if header:\n",
+    "            custom_dataset_len -= 1\n",
+    "\n",
+    "        # Optionally map the dataset labels using the map_function\n",
+    "        if map_function:\n",
+    "            custom_dataset = custom_dataset.map(map_function)\n",
+    "        \n",
+    "        # Create batches based on the specified batch size\n",
+    "        custom_dataset = custom_dataset.batch(batch_size)\n",
+    "\n",
+    "\n",
+    "        # Calculate sizes for the splits\n",
+    "        total_num_batches = int(custom_dataset_len / batch_size)\n",
+    "        train_size = int(train_percent * total_num_batches)\n",
+    "        val_size = int(val_percent * total_num_batches)\n",
+    "        test_size = int(test_percent * total_num_batches)\n",
+    "\n",
+    "        # Create the train, validation, and test splits\n",
+    "        self.train_ds = custom_dataset.take(train_size)    \n",
+    "        self.val_ds = custom_dataset.skip(train_size).take(val_size)\n",
+    "        self.test_ds = custom_dataset.skip(train_size).skip(val_size)\n",
+    "\n",
+    "        # Set the cardinality so that progress bars will work properly\n",
+    "        self.train_ds = self.train_ds.apply(tf.data.experimental.assert_cardinality(train_size))\n",
+    "        self.val_ds = self.val_ds.apply(tf.data.experimental.assert_cardinality(val_size))\n",
+    "        self.test_ds = self.test_ds.apply(tf.data.experimental.assert_cardinality(test_size))\n",
+    "\n",
+    "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
+    "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
+    "dataset_url = \"https://raw.githubusercontent.com/susanli2016/NLP-with-Python/master/data/title_conference.csv\"\n",
+    "dataset_directory = os.path.join(dataset_directory, \"titleconference\")\n",
+    "csv_name = \"title_conference.csv\"\n",
+    "delimiter = \",\"\n",
+    "header = True  # Set to true if the csv file has a header row\n",
+    "csv_path = os.path.join(dataset_directory, csv_name)\n",
+    "\n",
+    "if not os.path.exists(dataset_directory):\n",
+    "    os.makedirs(dataset_directory)\n",
+    "\n",
+    "# If we don't already have the csv file, download and extract the zip file to get it.\n",
+    "if not os.path.exists(csv_path):\n",
+    "    df = pd.read_csv(dataset_url, header=0)\n",
+    "    df.to_csv(csv_path, index=False)\n",
+    "\n",
+    "label_map = {\n",
+    "    \"VLDB\": 0,\n",
+    "    \"ISCAS\": 1,\n",
+    "    \"SIGGRAPH\": 2,\n",
+    "    \"INFOCOM\": 3, \n",
+    "    \"WWW\": 4\n",
+    "}\n",
+    "\n",
+    "int_to_label_map ={}\n",
+    "for k, v in label_map.items():\n",
+    "    int_to_label_map[v] = k\n",
+    "\n",
+    "# Map function to translate labels in the csv file to numerical values when loading the dataset\n",
+    "def map_title(features, label):\n",
+    "    label = tf.py_function(lambda x: label_map[x.numpy().decode('utf-8')], [label], tf.int64)\n",
+    "    return features, label\n",
+    "\n",
+    "# Initialize the dataset splits using the custom dataset\n",
+    "dataset = CustomCsvMultiTextClassificationData(csv_file=csv_path,\n",
+    "                                                delimiter=delimiter,\n",
+    "                                                header=header,\n",
+    "                                                train_percent=0.8,\n",
+    "                                                val_percent=0.1,\n",
+    "                                                test_percent=0.1,\n",
+    "                                                batch_size=batch_size,\n",
+    "                                                label_map=int_to_label_map,\n",
+    "                                                dataset_name=csv_name,\n",
+    "                                                map_function=map_title)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Build the model\n",
+    "\n",
+    "Create the BERT model to fine tune using a input layer, the preprocessing layer (from TF Hub), the BERT encoder layer (from TF Hub), one dense layer, and a dropout layer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_layer = tf.keras.layers.Input(shape=(), dtype=tf.string, name='input_layer')\n",
+    "preprocessing_layer = hub.KerasLayer(tfhub_preprocess, name='preprocessing')\n",
+    "encoder_inputs = preprocessing_layer(input_layer)\n",
+    "encoder_layer = hub.KerasLayer(tfhub_bert_encoder, trainable=True, name='encoder')\n",
+    "outputs = encoder_layer(encoder_inputs)\n",
+    "net = outputs['pooled_output']\n",
+    "net = tf.keras.layers.Dense(16, activation='relu', name='fully_connected_layer')(net)\n",
+    "# Add dropout layer for regularization\n",
+    "net = tf.keras.layers.Dropout(0.2)(net)\n",
+    "net = tf.keras.layers.Dense(len(label_map), activation='softmax', name='classifier')(net)\n",
+    "classifier_model = tf.keras.Model(input_layer, net)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Fine tuning and evaluation\n",
+    "\n",
+    "Train the model for the specified number of epochs, then evaluate the model using the test dataset."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Note that there is a known error during custom dataset training: `train_function (Empty logs). Please use Model.compile(..., run_eagerly=True), or tf.config.run_functions_eagerly(True) for more information of where went wrong, or file a issue/bug to tf.keras.`\n",
+    "> If you see this error, try using the first dataset for at least partially training (it doesn't have to finish). Then re-run with the custom dataset and training should work. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%time\n",
+    "\n",
+    "# The number of training epochs to run\n",
+    "num_train_epochs = 1\n",
+    "\n",
+    "# Learning rate\n",
+    "learning_rate = 3e-5\n",
+    "\n",
+    "# Maximum total input sequence length after WordPiece tokenization (longer sequences will be truncated)\n",
+    "max_seq_length = 128\n",
+    "\n",
+    "classifier_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5), \n",
+    "                        loss='sparse_categorical_crossentropy', \n",
+    "                        metrics=['accuracy'], run_eagerly=True)\n",
+    "\n",
+    "history = classifier_model.fit(dataset.train_ds,\n",
+    "                               validation_data=dataset.val_ds,\n",
+    "                               epochs=num_train_epochs)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Evaluate the accuracy using the test dataset. If the accuracy does not meet your expectations, try to increasing the size of the training dataset split or the number of training epochs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loss, accuracy = classifier_model.evaluate(dataset.test_ds)\n",
+    "\n",
+    "print(f'Loss: {loss}')\n",
+    "print(f'Accuracy: {accuracy}')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Predict using a single batch from the test dataset, and then display the results along with the input text and the actual label."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "num_steps = 1\n",
+    "predictions = classifier_model.predict(dataset.test_ds, batch_size=batch_size, steps=num_steps)\n",
+    "\n",
+    "prediction_list = []\n",
+    "step_count = 0\n",
+    "\n",
+    "for batch in dataset.test_ds:\n",
+    "    label_list = list(batch[1].numpy())\n",
+    "    text_list = list(batch[0].numpy())\n",
+    "    \n",
+    "    for i, (text, actual_label) in enumerate(zip(text_list, label_list)):\n",
+    "        score = tf.nn.softmax(predictions[i])\n",
+    "        score = tf.reduce_max(score)\n",
+    "        prediction = tf.math.argmax(predictions[i]).numpy()\n",
+    "        prediction = dataset.get_str_label(prediction)\n",
+    "        prediction_list.append([text.decode('utf-8'),\n",
+    "                                tf.get_static_value(score),\n",
+    "                                prediction,\n",
+    "                                dataset.get_str_label(actual_label)])\n",
+    "    \n",
+    "    step_count += 1\n",
+    "    if num_steps <= step_count:\n",
+    "        break\n",
+    "    \n",
+    "result_df = pd.DataFrame(prediction_list, columns=[\"Input Text\", \"Score\", \"Predicted Label\", \"Actual Label\"])\n",
+    "result_df.style.hide(axis=\"index\")\n",
+    "\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Export the model\n",
+    "\n",
+    "Since training has completed, export the `saved_model.pb` to the output directory in a folder with the model and dataset name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_dir = \"{}_{}\".format(model_name, dataset.dataset_name)\n",
+    "model_dir = os.path.join(output_directory, model_dir)\n",
+    "classifier_model.save(model_dir, include_optimizer=False)\n",
+    "\n",
+    "saved_model_path = os.path.join(model_dir, \"saved_model.pb\")\n",
+    "if os.path.exists(saved_model_path):\n",
+    "    print(\"Saved model location:\", saved_model_path)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Reload the model and make predictions\n",
+    "\n",
+    "Reload from the `saved_model.pb` in the output directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reloaded_model = tf.saved_model.load(model_dir)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The next section defines a list of strings to send as input to the reloaded model. If you are using a dataset other than the [AG News dataset](https://www.tensorflow.org/datasets/catalog/ag_news_subset), you can update the snippet below with your own list of input text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "input_text = [\"WASHINGTON - Employers stepped up hiring in August, expanding payrolls by 144,000 and lowering the unemployment rate to 5.4 percent.\",\n",
+    "              \"PRESENTACION, Philippines (Reuters) - Philippine communist rebels freed Wednesday two soldiers they had held as 'prisoners of war' for'\\\n",
+    "              'more than five months, saying they wanted to rebuild confidence in peace talks with the government.\", \n",
+    "              \"Geneva - Worldwide sales of industrial robots surged to record levels in the first half of 2004 after equipment prices fell while labour' \\\n",
+    "              'costs grew, the United Nations Economic Commission for Europe said in a report to be released today.\"]\n",
+    "    \n",
+    "if not input_text:\n",
+    "    raise ValueError(\"Please define the list of input_text strings.\")\n",
+    "\n",
+    "# Send the input text to the reloaded model\n",
+    "predict_results = tf.nn.softmax(reloaded_model(tf.constant(input_text)))\n",
+    "\n",
+    "# Get the results into a data frame to display\n",
+    "result_list = [[input_text[i],\n",
+    "                tf.get_static_value(tf.reduce_max(predict_results[i])),\n",
+    "                dataset.get_str_label(tf.math.argmax(predict_results[i]))] for i in range(len(input_text))]\n",
+    "result_df = pd.DataFrame(result_list, columns=[\"Input Text\", \"Score\", \"Predicted Label\"])\n",
+    "result_df.style.hide(axis=\"index\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Citations\n",
+    "\n",
+    "```\n",
+    "@misc{zhang2015characterlevel,\n",
+    "    title={Character-level Convolutional Networks for Text Classification},\n",
+    "    author={Xiang Zhang and Junbo Zhao and Yann LeCun},\n",
+    "    year={2015},\n",
+    "    eprint={1509.01626},\n",
+    "    archivePrefix={arXiv},\n",
+    "    primaryClass={cs.LG}\n",
+    "}\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "tlt_notebook_venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "e0aa4059be174cbf31be2e5e13e301e374cc50c7151267adca3135ebe59561bd"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/README.md b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/README.md
new file mode 100644
index 000000000..210e6ed82
--- /dev/null
+++ b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/README.md
@@ -0,0 +1,50 @@
+# Text Classifier fine tuning with TensorFlow
+
+These notebooks demonstrate fine tuning using various [BERT](https://arxiv.org/abs/1810.04805) models
+from [TF Hub](https://tfhub.dev) using Intel® Optimization for TensorFlow for text classification.
+
+The notebook performs the following steps:
+1. Install dependencies and setup parameters
+1. Prepare the dataset
+1. Build the model
+1. Fine tuning and evaluation
+1. Export the model
+1. Reload the model and make predictions
+
+## Running the notebook
+
+To run the notebook, follow the instructions in `setup.md`.
+
+## References
+
+Dataset citations:
+```
+@InProceedings{maas-EtAl:2011:ACL-HLT2011,
+  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
+  title     = {Learning Word Vectors for Sentiment Analysis},
+  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
+  month     = {June},
+  year      = {2011},
+  address   = {Portland, Oregon, USA},
+  publisher = {Association for Computational Linguistics},
+  pages     = {142--150},
+  url       = {http://www.aclweb.org/anthology/P11-1015}
+}
+
+@misc{zhang2015characterlevel,
+    title={Character-level Convolutional Networks for Text Classification},
+    author={Xiang Zhang and Junbo Zhao and Yann LeCun},
+    year={2015},
+    eprint={1509.01626},
+    archivePrefix={arXiv},
+    primaryClass={cs.LG}
+}
+
+@misc{misc_sms_spam_collection_228,
+  author       = {Almeida, Tiago},
+  title        = {{SMS Spam Collection}},
+  year         = {2012},
+  howpublished = {UCI Machine Learning Repository}
+}
+```
+Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.
\ No newline at end of file
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/bert_utils.py b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/bert_utils.py
similarity index 76%
rename from docs/notebooks/transfer_learning/tfhub_bert/bert_utils.py
rename to docs/notebooks/transfer_learning/tfhub_bert/text_classification/bert_utils.py
index 78d634b3a..f361687d1 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/bert_utils.py
+++ b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/bert_utils.py
@@ -17,11 +17,8 @@
 #
 
 import json
-import os
 import pandas as pd
 
-from zipfile import ZipFile
-
 
 def get_model_map(json_path, return_data_frame=False):
     """
@@ -55,23 +52,3 @@ def get_model_map(json_path, return_data_frame=False):
         return tfhub_model_map, models_df
     else:
         return tfhub_model_map
-
-
-def download_and_extract_zip(zip_url, destination_dir):
-    """
-    Downloads and extracts a zip file to the specified destination.
-    """
-    if not os.path.exists(destination_dir):
-        os.makedirs(destination_dir)
-
-    zip_filename = os.path.basename(zip_url)
-
-    # Download the zip file
-    if not os.path.exists(os.path.join(destination_dir, zip_filename)):
-        import wget
-        wget.download(zip_url, destination_dir)
-
-    # Extract the zip file
-    dataset_zip_path = os.path.join(destination_dir, zip_filename)
-    with ZipFile(dataset_zip_path, "r") as zipfile:
-        zipfile.extractall(path=destination_dir)
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/tfhub_bert_model_map_classifier.json b/docs/notebooks/transfer_learning/tfhub_bert/text_classification/tfhub_bert_model_map_classifier.json
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/tfhub_bert_model_map_classifier.json
rename to docs/notebooks/transfer_learning/tfhub_bert/text_classification/tfhub_bert_model_map_classifier.json
diff --git a/k8s/common/tensorflow/KubernetesDevOps.md b/k8s/common/tensorflow/KubernetesDevOps.md
deleted file mode 100644
index 946e2d662..000000000
--- a/k8s/common/tensorflow/KubernetesDevOps.md
+++ /dev/null
@@ -1,229 +0,0 @@
-# Kubernetes Dev Ops
-
-## MPI Operator Deployment
-
-The following instructions explain how to deploy the [MPI Operator](https://github.com/kubeflow/mpi-operator/)
-to a Kubernetes cluster. This deployment needs to be done by a devops user with Cluster RBAC permissions.
-
-We have a [copy of the `mpi-operator.yaml`](/tools/k8s/devops/operators/mpi-operator/mpi-operator.yaml)
-that is compatible with our multi-node Kubernetes packages. There are instructions on how to deploy this
-mpi-operator in this [README](/tools/k8s/devops/operators/mpi-operator/README.md).
-
-If you prefer to install it from the [MPI Operator repo](https://github.com/kubeflow/mpi-operator/),
-follow the steps below that explain how to deploy the mpi-operator on your cluster using version `v0.2.3`.
-
-1. Download the v1alpha2 `mpi-operator.yaml` file from the v0.2.3 tag
-   ```
-   curl -o mpi-operator.yaml https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.2.3/deploy/v1alpha2/mpi-operator.yaml
-   ```
-
-2. Change line #189 from `image: mpioperator/mpi-operator:latest` to `image: mpioperator/mpi-operator:v0.2.3`
-
-3. Deploy the operator
-   ```
-   kubectl apply -f mpi-operator.yaml
-   ```
-
-For more information on deploying the mpi-operator to the k8s cluster, see the
-[documentation in the mpi-operator repo](https://github.com/kubeflow/mpi-operator#mpi-operator).
-
-## Argo Deployment
-
-Prior to running Argo workflows, the controller needs to be deployed on
-the cluster. See [Argo's Quick Start Guide](https://github.com/argoproj/argo-workflows/blob/stable/docs/quick-start.md)
-for installation instructions.
-
-Running the [hello world workflow](https://github.com/argoproj/argo-workflows/blob/stable/docs/quick-start.md)
-and verifying that it succeeds is a good test to verify that the argo
-installation is working. Using the Argo CLI is optional. The argo yaml
-can be deployed using `kubectl create`.
-
-## Enabling CPU Pinning in model workflows
-
-Models running single or multi-node workflows on kubernetes-v1.18+ can claim exclusive use of cpus within a single numa node by 
-leveraging the _native_ kubernetes [Topology Manager](https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/). 
-More sophisticated cpu policies may be enabled by installing the [CRI Resource Manager](https://intel.github.io/cri-resource-manager/stable/docs/index.html).
-This additional capability will not be discussed here, rather just the native kubernetes cpu-pinning - available in k8-v1.18 (beta).
-
-CPU Pinning on a kubernetes node is done by:
-
-1. Enabling the __Topology Manager__ on one or more nodes within a cluster.
-2. Specifying a Qos class of [Guaranteed](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed) within the workflow yaml. An example of a yaml file that specifies a Pod with a Guaranteed Qos class is shown below:
-
-```
-apiVersion: v1
-kind: Pod
-metadata:
-  name: cpu-pinning-test
-spec:
-  containers:
-  - name: cpu-pinning-test
-    image: bash
-    resources:
-      limits:
-        memory: "100M"
-        cpu: "2"
-      requests:
-        memory: "100M"
-        cpu: "2"
-    command: 
-    - sleep
-    args:
-    - infinity
-    imagePullPolicy: IfNotPresent
-```
-
-### Enabling the Topology Manager on a kubernetes node
-
-In order to use the topology manager, the kubernetes node must be running a kubelet with the startup argument [--dynamic-config-dir](https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/#before-you-begin). This startup option allows the node's kubelet to be configured dynamically using a ConfigMap. Modifying a kubelet may disrupt existing pods running on that node, so operations that enable a node's cpu pinning should include cordoning the node and adding scheduling constraints. This sequence of operations is shown below. The operation sequence is then implemented within a bash script[^✝], annotated to match each operation step. 
-
-[^✝]: the bash script assumes [kubectl-v1.19+](https://github.com/kubernetes/kubectl/releases) and [yq-v3.4.1](https://github.com/mikefarah/yq/releases/tag/3.4.1) are installed.
-1. Get the node's kubelet configuration and add cpu-manager, topology-manager features
-2. Taint the node with [NoSchedule](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#example-use-cases) and [NoExecute](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions).
-3. Create a ConfigMap in the kube-system namespace that has the KubeletConfiguration data.
-4. Patch the node so that it uses the ConfigMap.
-5. Monitor the node to validate it has recognized and is using the ConfigMap.
-6. [Drain](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain) the node of pods that aren't critical.
-7. Uncordon the node.
-8. Deploy a workflow yaml.
-9. Validate the workflow is using reserved cpu's within a single numa node.
-
-#### sequence diagram of operations
-
-```mermaid
-sequenceDiagram
-    kubectl->>+node: get kubelet config
-    kubectl->>+kubectl: save kubelet config and add cpu-manager, topology-manager features
-    kubectl->>+node: taint the node with NoSchedule, NoExecute
-    kubectl->>+configmap: create ConfigMap that has the KubeletConfiguration data
-    kubectl->>+node: patch the node so that it uses the ConfigMap
-    node->>+configmap: node fetches configmap and updates its status
-    node->>+node: updates its status
-    kubectl->>+node: monitor node's status 
-    kubectl->>+node: drain the node of jobs
-    kubectl->>+node: uncordon the node
-    kubectl->>+node: deploy a workflow yaml
-    kubectl->>+node: validate the workflow is using cpus in the single-numa-node
-```
-
-#### bash script: enable-cpu-pinning.sh
-
-```bash
-#
-# calls enable-cpu-pinning with a node name
-#
-if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
-  node.enable-cpu-pinning $@
-fi
-```
-
-#### bash function: _get-kubelet-config_
-
-```bash
-#
-# get the node's kubelet configuration
-# Reference: [Generate the configuration file](https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/#generate-the-configuration-file)
-#
-# Note: add to bash script enable-cpu-pinning.sh
-#
-node.get-kubelet-config()
-{
-  local _node _proxy_pid _tmp_file1
-  if (( $# != 1 )); then
-    echo 'usage: '${FUNCNAME[0]##*.}' <node>' >&2
-    return
-  fi
-  _node=$!
-  kubectl proxy --port=8001 &
-  _proxy_pid=$!
-  _tmp_file1=$(mktemp /tmp/node.kubelet-config-override.XXXXXXX)
-  cat > $_tmp_file1 <<KUBELET_OVERRIDES
-kind: KubeletConfiguration
-apiVersion: kubelet.config.k8s.io/v1beta1
-cpuManagerPolicy: static
-topologyManagerPolicy: single-numa-node
-topologyManagerScope: pod
-featureGates:
-  CPUManager: true
-  TopologyManager: true
-kubeReserved:
-  cpu: "1"
-systemReserved:
-  cpu: "1"
-KUBELET_OVERRIDES
-  curl -sSL "http://localhost:8001/api/v1/nodes/node2/proxy/configz" | yq r - kubeletconfig --prettyPrint | yq m -x - $_tmp_file1 > /tmp/kubelet_configz.yaml
-  kill $_proxy_pid 2>&1 >/dev/null
-  rm -f $_tmp_file1
-}
-```
-
-#### bash function: _get-status-config_
-
-```bash
-#
-# polls the node until it updates its status.config
-#
-# Note: add to bash script enable-cpu-pinning.sh
-#
-node.get-status-config()
-{
-  local _node _output
-  if (( $# != 1 )); then
-    echo 'usage: '${FUNCNAME[0]##*.}' <node>' >&2
-    return
-  fi
-  _node=$1
-  _output='{}'
-  while (( ${#_output} == 2 )); do
-    sleep 2
-    echo 'Waiting for '$_node' to update status.config'
-    _output=$(kubectl get node $_node -oyaml | yq r - status.config)
-  done
-  kubectl get node $_node -oyaml | yq r - status.config
-}
-```
-
-#### bash function: _enable-cpu-pinning_
-
-```bash
-#
-# gets a node's kubelet config
-# adds topology-manager=single-numa-node, cpu-manager=static
-# creates a configmap
-# patches the node's to use the configmap
-#
-# Note: add to bash script enable-cpu-pinning.sh
-#
-node.enable-cpu-pinning()
-{
-  local _node_cm _node _patch
-  if (( $# != 1 )); then
-    echo 'usage: '${FUNCNAME[0]##*.}' <node>' >&2
-    return
-  fi
-  _node=$1
-  _node_cm=$(kubectl -n kube-system get cm -oname | grep ${_node}-config)
-  _node_cm=${_node_cm#configmap/}
-  if [[ -n $_node_cm ]]; then
-    kubectl -n kube-system delete cm $_node_cm
-  fi
-  rm -f /tmp/kubelet_configz.yaml
-  node.get-kubelet-config $_node
-  #Step 1.
-  kubectl taint nodes $_node dedicated=cpu-pinning:NoSchedule
-  kubectl taint nodes $_node dedicated=cpu-pinning:NoExecute
-  #Step 2.
-  kubectl -n kube-system create configmap ${_node}-config --from-file=kubelet=/tmp/kubelet_configz.yaml --append-hash
-  _node_cm=$(kubectl -n kube-system get cm -oname | grep ${_node}-config)
-  _node_cm=${_node_cm#configmap/}
-  _patch=$'{"spec":{"configSource":{"configMap":{"name":"'$_node_cm$'","namespace":"kube-system","kubeletConfigKey":"kubelet"}}}}'
-  #Step 3.
-  kubectl patch node $_node -p "$_patch"
-  #Step 4.
-  node.get-status-config $_node
-  #Step 5.
-  kubectl drain $_node --ignore-daemonsets &&
-  #Step 6.
-  kubectl uncordon $_node
-}
-```
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/datasets.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/datasets.md
deleted file mode 100644
index f8b3437b3..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-OPTIONAL: The ImageNet dataset can be used in these <model name> Kubernetes examples.
-You can download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/description.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/description.md
deleted file mode 100644
index 07a043dac..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/description.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/download.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/kubernetes.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/kubernetes.md
deleted file mode 100644
index 48fcf6151..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/kubernetes.md
+++ /dev/null
@@ -1,182 +0,0 @@
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the ResNet50 v1.5 FP32 inference package.
-```
-wget <package url>
-tar -xvf <package name>
-```
-
-### Execution
-
-The Kubernetes* package for `<model name> <precision> <mode>` includes a kubernetes serving deployment.
-The directory tree within the model package is shown below, where the serving directory is below the
-[mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-
-```
-quickstart
-└── mlops
-    └── serving
-```
-
-#### Prerequisites
-
-The <package name> package uses [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure parameters within the deployment.yaml. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### TensorFlow Serving
-
-The multi-node serving use case makes the following assumptions:
-- the model directory is a shared volume mounted cluster wide
-- a saved model has been generated and saved under the model directory volume
-
-TensorFlow Serving is run by submitting a deployment and service yaml file to the k8s api-server,
-which results in the creation of pod replicas, each serving the model on a different node of the cluster.
-
-The steps follow the
-[TensorFlow Serving with Kubernetes instructions](https://www.tensorflow.org/tfx/serving/serving_kubernetes)
-with the exception that it does not use a Google Cloud Kubernetes
-cluster. Since the Kubernetes cluster being used does not have a load
-balancer, the configuration is setup for NodePort, which will allow
-external requests.
-
-Make sure you are inside the serving directory:
-
-```
-cd <package dir>/quickstart/mlops/serving
-```
-
-The parameters that should be changed within the serving resources are shown in the table below:
-
-|            NAME             |                  VALUE                   |         DESCRIPTION         |
-|-----------------------------|------------------------------------------|-----------------------------|
-| FS_ID                       | 0                                        | owner id of mounted volumes |
-| GROUP_ID                    | 0                                        | process group id            |
-| GROUP_NAME                  | root                                     | process group name          |
-| MODEL_BASE_PATH             | /models                                  | mounted model directory     |
-| MODEL_DIR                   | /models                                  | host model base directory   |
-| MODEL_NAME                  | resnet50v1_5                             | model name                  |
-| MODEL_PORT                  | 8500                                     | model container port        |
-| MODEL_SERVICE_PORT          | 8501                                     | model service port          |
-| REPLICAS                    | 3                                        | number of replicas          |
-| USER_ID                     | 0                                        | process owner id            |
-| USER_NAME                   | root                                     | process owner name          |
-
-
-For example to change the MODEL_SERVICE_PORT from 8500 to 9500
-
-```
-kustomize cfg set . MODEL_SERVICE_PORT 9500
-```
-
-The user should change the values below so the pod is deployed with the user's identity, rather than running as root.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the values below so the SavedModel is mounted into the pod from the host machine.
-
-```
-kustomize cfg set . MODEL_BASE_PATH <Full path in the pod where the model directory will be mounted> -R
-kustomize cfg set . MODEL_DIR <Full path on the host to the directory containing the model directory> -R
-kustomize cfg set . MODEL_NAME <Model name - must be the name of the directory containing the SavedModel> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-kustomize edit set namespace <User's namespace>
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the <package name> by running:
-
-```
-kustomize build  . | kubectl apply -f -
-```
-
-##### TensorFlow Serving Client
-
-Once all the pods are running, the TensorFlow
-Serving gRPC client can be used to run inference on the served model.
-
-Prior to running the client script, install the following dependency in
-your environment:
-* tensorflow-serving-api
-
-The client script accepts an optional argument `--data_dir` pointing to a directory of images in TF records format. 
-If the argument is not present, dummy data will be used instead. Then the script formats the data as a gRPC request object and
-calls the served model API. Benchmarking metrics are printed out.
-
-Run the [run_tf_serving_client.py](run_tf_serving_client.py) script with
-the `--help` flag to see the argument options:
-```
-$ python <package name>/quickstart/run_tf_serving_client.py --help
-Send TF records or simulated image data to tensorflow_model_server loaded with ResNet50v1_5 model.
-
-flags:
-
-run_tf_serving_client.py:
-  --batch_size: Batch size to use
-    (default: '1')
-    (an integer)
-  --data_dir: Path to images in TF records format
-    (default: '')
-  --server: PredictionService host:port
-    (default: 'localhost:8500')
-```
-
-1. Find the `INTERNAL-IP` of one of the nodes in your cluster using
-   `kubectl get nodes -o wide`. This IP should be used as the server URL
-   in the `--server` arg.
-
-1. Get the `NodePort` using `kubectl describe service`. This `NodePort`
-   should be used as the port in the `--server` arg.
-
-1. Run the client script with your preferred parameters. For example:
-   ```
-   python <package name>/quickstart/run_tf_serving_client.py --server <Internal IP>:<Node Port> --data_dir <path to TF records> --batch_size <batch size>
-   ```
-   The script will call the served model using data from the `data_dir` path or simulated data
-   and output performance metrics.
-   
-1. If you want to send multiple parallel calls to the server, you can use the `multi_client.sh` script.
-   Its options are:
-   ```
-   --model: Model to be inferenced
-   --batch_size: Number of samples per batch (default 1)
-   --servers: "server1:port;server2:port" (default "localhost:8500")
-   --clients: Number of clients per server (default 1)
-   ```
-   
-   You must be in the quickstart folder to run it, and the script does not currently support a `data_dir` argument (only runs with simulated data).
-   Example:
-   
-   ```
-   cd <package name>/quickstart
-   bash multi_client.sh --model=resnet50v1_5 --servers="<Internal IP>:<Node Port>" --clients=5
-   cd ../../
-   ```
-
-##### Clean up the pipeline
-
-To clean up the served model, delete the service,
-deployment, and other resources using the following commands:
-```
-kustomize build  . | kubectl delete -f -
-```
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/license.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/quickstart.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/quickstart.md
deleted file mode 100644
index 19f85678c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`run_tf_serving_client.py`](run_tf_serving_client.py) | Runs gRPC client for multi-node batch and online inference. |
-| [`multi_client.sh`](multi_client.sh) | Runs multiple parallel gRPC clients for multi-node batch and online inference. |
-
-These quickstart scripts can be run in this environment:
-* [Kubernetes](#kubernetes)
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/title.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/troubleshooting.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/troubleshooting.md
deleted file mode 100644
index fd80903fb..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.docs/troubleshooting.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/deployments-yml-exist.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/deployments-yml-exist.sh
deleted file mode 100644
index 0af5c5ecc..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/deployments-yml-exist.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-load "lib/test_helper"
-
-@test "validate package for resnet50v1-5-fp32-inference-k8s in framework k8s creates model package file" {
-  run model-builder package -f k8s resnet50v1-5-fp32-inference-k8s
-  (( $status == 0 ))
-  [[ -f output/resnet50v1-5-fp32-inference-k8s.tar.gz ]]
-}
-
-@test "validate generate-deployment for resnet50v1-5-fp32-inference-k8s in framework k8s creates deployment files under deployments/resnet50v1-5-fp32-training-k8s/" {
-  run model-builder generate-deployment -f k8s resnet50v1-5-fp32-inference-k8s
-  (( $status == 0 ))
-  [[ -f deployments/resnet50v1-5-fp32-inference-k8s/serving/serving.yaml ]]
-  (( $(last_modified deployments/resnet50v1-5-fp32-inference-k8s/serving/serving.yaml) <= 50 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/namespace-exists.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/namespace-exists.sh
deleted file mode 100644
index 4a76a6981..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/namespace-exists.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "verify namespace $USER_NAME exists" {
-  run verify "there is 1 namespace named '"$USER_NAME"'"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/serving_deployment.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/serving_deployment.sh
deleted file mode 100644
index 95b1a0c1c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/.tests/serving_deployment.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "kubectl apply of serving resnet50v1-5-fp32-inference-k8s" {
-  run $DETIK_CLIENT_NAME apply -f deployments/resnet50v1-5-fp32-inference-k8s/serving/serving.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "deployment of resnet50v1-5-fp32-inference-k8s serving creates 1 pod and it is running" {
-  try "at most 4 times every 30s to get pods named 'resnet50v1-5-fp32-inference' and verify that 'status' is 'running'"
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "verify job completion, check end of output includes text 'run_final'" {
-  run launcher_logs
-  echo "output = ${output}"
-  (( $status == 0 ))
-  [[ ${lines[4]} =~ run_final ]]
-}
-
-@test "kubectl delete of resnet50v1-5-fp32-inference-k8s serving" {
-  run $DETIK_CLIENT_NAME delete -f deployments/resnet50v1-5-fp32-inference-k8s/serving/serving.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/README.md b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/README.md
deleted file mode 100644
index a89920b33..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/README.md
+++ /dev/null
@@ -1,241 +0,0 @@
-<!--- 0. Title -->
-# ResNet50 v1.5 FP32 inference
-
-<!-- 10. Description -->
-
-This document has instructions for running ResNet50 v1.5 FP32 inference using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
-
-<!--- 20. Download link -->
-## Download link
-
-[resnet50v1-5-fp32-inference-k8s.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/resnet50v1-5-fp32-inference-k8s.tar.gz)
-
-<!--- 30. Datasets -->
-## Dataset
-
-OPTIONAL: The ImageNet dataset can be used in these ResNet50 v1.5 Kubernetes examples.
-You can download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`run_tf_serving_client.py`](run_tf_serving_client.py) | Runs gRPC client for multi-node batch and online inference. |
-| [`multi_client.sh`](multi_client.sh) | Runs multiple parallel gRPC clients for multi-node batch and online inference. |
-
-These quickstart scripts can be run in this environment:
-* [Kubernetes](#kubernetes)
-
-
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the ResNet50 v1.5 FP32 inference package.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/resnet50v1-5-fp32-inference-k8s.tar.gz
-tar -xvf resnet50v1-5-fp32-inference-k8s.tar.gz
-```
-
-### Execution
-
-The Kubernetes* package for `ResNet50 v1.5 FP32 inference` includes a kubernetes serving deployment.
-The directory tree within the model package is shown below, where the serving directory is below the
-[mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-
-```
-quickstart
-└── mlops
-    └── serving
-```
-
-#### Prerequisites
-
-The resnet50v1-5-fp32-inference-k8s.tar.gz package uses [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure parameters within the deployment.yaml. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### TensorFlow Serving
-
-The multi-node serving use case makes the following assumptions:
-- the model directory is a shared volume mounted cluster wide
-- a saved model has been generated and saved under the model directory volume
-
-TensorFlow Serving is run by submitting a deployment and service yaml file to the k8s api-server,
-which results in the creation of pod replicas, each serving the model on a different node of the cluster.
-
-The steps follow the
-[TensorFlow Serving with Kubernetes instructions](https://www.tensorflow.org/tfx/serving/serving_kubernetes)
-with the exception that it does not use a Google Cloud Kubernetes
-cluster. Since the Kubernetes cluster being used does not have a load
-balancer, the configuration is setup for NodePort, which will allow
-external requests.
-
-Make sure you are inside the serving directory:
-
-```
-cd resnet50v1-5-fp32-inference-k8s/quickstart/mlops/serving
-```
-
-The parameters that should be changed within the serving resources are shown in the table below:
-
-|            NAME             |                  VALUE                   |         DESCRIPTION         |
-|-----------------------------|------------------------------------------|-----------------------------|
-| FS_ID                       | 0                                        | owner id of mounted volumes |
-| GROUP_ID                    | 0                                        | process group id            |
-| GROUP_NAME                  | root                                     | process group name          |
-| MODEL_BASE_PATH             | /models                                  | mounted model directory     |
-| MODEL_DIR                   | /models                                  | host model base directory   |
-| MODEL_NAME                  | resnet50v1_5                             | model name                  |
-| MODEL_PORT                  | 8500                                     | model container port        |
-| MODEL_SERVICE_PORT          | 8501                                     | model service port          |
-| REPLICAS                    | 3                                        | number of replicas          |
-| USER_ID                     | 0                                        | process owner id            |
-| USER_NAME                   | root                                     | process owner name          |
-
-
-For example to change the MODEL_SERVICE_PORT from 8500 to 9500
-
-```
-kustomize cfg set . MODEL_SERVICE_PORT 9500
-```
-
-The user should change the values below so the pod is deployed with the user's identity, rather than running as root.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the values below so the SavedModel is mounted into the pod from the host machine.
-
-```
-kustomize cfg set . MODEL_BASE_PATH <Full path in the pod where the model directory will be mounted> -R
-kustomize cfg set . MODEL_DIR <Full path on the host to the directory containing the model directory> -R
-kustomize cfg set . MODEL_NAME <Model name - must be the name of the directory containing the SavedModel> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-kustomize edit set namespace <User's namespace>
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the resnet50v1-5-fp32-inference-k8s.tar.gz by running:
-
-```
-kustomize build  . | kubectl apply -f -
-```
-
-##### TensorFlow Serving Client
-
-Once all the pods are running, the TensorFlow
-Serving gRPC client can be used to run inference on the served model.
-
-Prior to running the client script, install the following dependency in
-your environment:
-* tensorflow-serving-api
-
-The client script accepts an optional argument `--data_dir` pointing to a directory of images in TF records format. 
-If the argument is not present, dummy data will be used instead. Then the script formats the data as a gRPC request object and
-calls the served model API. Benchmarking metrics are printed out.
-
-Run the [run_tf_serving_client.py](run_tf_serving_client.py) script with
-the `--help` flag to see the argument options:
-```
-$ python resnet50v1-5-fp32-inference-k8s.tar.gz/quickstart/run_tf_serving_client.py --help
-Send TF records or simulated image data to tensorflow_model_server loaded with ResNet50v1_5 model.
-
-flags:
-
-run_tf_serving_client.py:
-  --batch_size: Batch size to use
-    (default: '1')
-    (an integer)
-  --data_dir: Path to images in TF records format
-    (default: '')
-  --server: PredictionService host:port
-    (default: 'localhost:8500')
-```
-
-1. Find the `INTERNAL-IP` of one of the nodes in your cluster using
-   `kubectl get nodes -o wide`. This IP should be used as the server URL
-   in the `--server` arg.
-
-1. Get the `NodePort` using `kubectl describe service`. This `NodePort`
-   should be used as the port in the `--server` arg.
-
-1. Run the client script with your preferred parameters. For example:
-   ```
-   python resnet50v1-5-fp32-inference-k8s.tar.gz/quickstart/run_tf_serving_client.py --server <Internal IP>:<Node Port> --data_dir <path to TF records> --batch_size <batch size>
-   ```
-   The script will call the served model using data from the `data_dir` path or simulated data
-   and output performance metrics.
-   
-1. If you want to send multiple parallel calls to the server, you can use the `multi_client.sh` script.
-   Its options are:
-   ```
-   --model: Model to be inferenced
-   --batch_size: Number of samples per batch (default 1)
-   --servers: "server1:port;server2:port" (default "localhost:8500")
-   --clients: Number of clients per server (default 1)
-   ```
-   
-   You must be in the quickstart folder to run it, and the script does not currently support a `data_dir` argument (only runs with simulated data).
-   Example:
-   
-   ```
-   cd resnet50v1-5-fp32-inference-k8s.tar.gz/quickstart
-   bash multi_client.sh --model=resnet50v1_5 --servers="<Internal IP>:<Node Port>" --clients=5
-   cd ../../
-   ```
-
-##### Clean up the pipeline
-
-To clean up the served model, delete the service,
-deployment, and other resources using the following commands:
-```
-kustomize build  . | kubectl delete -f -
-```
-
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/Krmfile b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/Krmfile
deleted file mode 100644
index b19c64889..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/Krmfile
+++ /dev/null
@@ -1,187 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: model name
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: resnet50v1_5
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SERVING_NAME:
-      description: model serving name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SERVING_NAME
-          value: resnet50v1-5-fp32-inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SERVING_LABEL:
-      description: selector label
-      x-k8s-cli:
-        setter:
-          name: MODEL_SERVING_LABEL
-          value: resnet50v1-5-fp32-server
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SERVING_IMAGE_NAME:
-      description: image name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SERVING_IMAGE_NAME
-          value: intel/intel-optimized-tensorflow-serving
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SERVING_IMAGE_VERSION:
-      description: image tag
-      x-k8s-cli:
-        setter:
-          name: MODEL_SERVING_IMAGE_VERSION
-          value: latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/${MODEL_SERVING_IMAGE_NAME}:${MODEL_SERVING_IMAGE_VERSION}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${MODEL_SERVING_IMAGE_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SERVING_IMAGE_NAME'
-          - marker: ${MODEL_SERVING_IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SERVING_IMAGE_VERSION'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: mounted model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /models
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.MODEL_BASE_PATH:
-      description: base directory name
-      x-k8s-cli:
-        setter:
-          name: MODEL_BASE_PATH
-          value: /models
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.MODEL_PORT:
-      type: integer
-      description: model container port
-      x-k8s-cli:
-        setter:
-          name: MODEL_PORT
-          value: "8500"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.MODEL_SERVICE_PORT:
-      type: integer
-      description: model service port
-      x-k8s-cli:
-        setter:
-          name: MODEL_SERVICE_PORT
-          value: "8501"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REPLICAS:
-      type: integer
-      description: number of replicas
-      x-k8s-cli:
-        setter:
-          name: REPLICAS
-          value: "3"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/config-map.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/config-map.yaml
deleted file mode 100644
index b8279a6f1..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/config-map.yaml
+++ /dev/null
@@ -1,14 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: resnet50v1-5-fp32-inference # {"$openapi":"MODEL_SERVING_NAME"}
-data:
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  MODEL_BASE_PATH: /models # {"$openapi":"MODEL_BASE_PATH"}
-  REPLICAS: "3"
-  TF_CPP_MIN_VLOG_LEVEL: "1"
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
-  MODEL_NAME: resnet50v1_5 # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/deployment.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/deployment.yaml
deleted file mode 100644
index c73b99872..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/deployment.yaml
+++ /dev/null
@@ -1,33 +0,0 @@
-apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
-kind: Deployment
-metadata:
-  name: resnet50v1-5-fp32-inference # {"$openapi":"MODEL_SERVING_NAME"}
-spec:
-  selector:
-    matchLabels:
-      app: resnet50v1-5-fp32-server # {"$openapi":"MODEL_SERVING_LABEL"}
-  replicas: 3 # {"$openapi":"REPLICAS"}
-  template:
-    metadata:
-      labels:
-        app: resnet50v1-5-fp32-server # {"$openapi":"MODEL_SERVING_LABEL"}
-    spec:
-      securityContext:
-        runAsUser: 0 # {"$openapi":"USER_ID"}
-        runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-        fsGroup: 0 # {"$openapi":"FS_ID"}
-      containers:
-      - name: model-server
-        image: docker.io/intel/intel-optimized-tensorflow-serving:latest # {"$openapi":"IMAGE"}
-        envFrom:
-        - configMapRef:
-            name: resnet50v1-5-fp32-inference # {"$openapi":"MODEL_SERVING_NAME"}
-        volumeMounts:
-        - name: models
-          mountPath: /models # {"$openapi":"MODEL_BASE_PATH"}
-        ports:
-        - containerPort: 8500 # {"$openapi":"MODEL_PORT"}
-      volumes:
-      - name: models
-        hostPath:
-          path: /models # {"$openapi":"MODEL_DIR"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/kustomization.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/kustomization.yaml
deleted file mode 100644
index cae38900c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service.yaml
-- deployment.yaml
-namespace: default
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/service.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/service.yaml
deleted file mode 100644
index d95293332..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/mlops/serving/service.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: resnet50v1-5-fp32-inference # {"$openapi":"MODEL_SERVING_NAME"}
-  labels:
-    app: resnet50v1-5-fp32-server # {"$openapi":"MODEL_SERVING_LABEL"}
-spec:
-  # comment or delete the following line if you want to use a LoadBalancer
-  type: NodePort # https://kubernetes.io/docs/concepts/services-networking/service/#nodeport
-  # if your cluster supports it, uncomment the following to automatically create
-  # an external load-balanced IP for the frontend service.
-  # type: LoadBalancer
-  ports:
-  - protocol: TCP
-    port: 8501 # {"$openapi":"MODEL_SERVICE_PORT"}
-    targetPort: 8500 # {"$openapi":"MODEL_PORT"}
-  selector:
-    app: resnet50v1-5-fp32-server # {"$openapi":"MODEL_SERVING_LABEL"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/multi_client.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/multi_client.sh
deleted file mode 100644
index 3b08b8d1c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/multi_client.sh
+++ /dev/null
@@ -1,65 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-PATTERN='[-a-zA-Z0-9_]*='
-if [ $# -eq 0 ] || [ $# -gt 4 ]; then
-    echo 'ERROR:'
-    echo "Expected 1-4 parameters got $#"
-    printf 'Please use following parameters:
-    --model=model to be inferenced
-    --batch_size=number of samples per batch (default 1)
-    --servers="server1:port;server2:port" (default "localhost:8500")
-    --clients=number of clients per server (default 1)
-    '
-    exit 1
-fi
-
-batch_size=1
-clients=1
-servers="localhost:8500"
-
-for i in "$@"
-do
-    case $i in
-        --model=*)
-            model=`echo $i | sed "s/${PATTERN}//"`;;
-	--batch_size=*)
-            batch_size=`echo $i | sed "s/${PATTERN}//"`;;
-        --servers=*)
-            servers=`echo $i | sed "s/${PATTERN}//"`;;
-        --clients=*)
-            clients=`echo $i | sed "s/${PATTERN}//"`;;
-        *)
-            echo "Parameter $i not recognized."; exit 1;;
-    esac
-done
-
-if [ -v ${model} ]; then
-    echo "Parameter model is required!";
-    exit 1
-fi
-
-server_list=($(echo "$servers" | tr ';' '\n'))
-
-for client in `seq ${clients}`
-do
-    for server in "${server_list[@]}"
-    do 
-        command="python run_tf_serving_client.py --model ${model} --batch_size ${batch_size} --server ${server}";
-	${command} &
-    done
-done
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/run_tf_serving_client.py b/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/run_tf_serving_client.py
deleted file mode 100644
index 620bb5e9d..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/inference/fp32/run_tf_serving_client.py
+++ /dev/null
@@ -1,137 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-"""Send TF records or simulated image data to tensorflow_model_server loaded with ResNet50v1_5 model.
-"""
-
-from __future__ import print_function
-
-import os
-import random
-
-import grpc
-import numpy as np
-import tensorflow as tf
-import tensorflow.compat.v1 as tf_v1
-import time
-from tensorflow_serving.apis import predict_pb2
-from tensorflow_serving.apis import prediction_service_pb2_grpc
-
-tf_v1.disable_eager_execution()
-
-tf_v1.app.flags.DEFINE_string('server', 'localhost:8500', 'PredictionService host:port')
-tf_v1.app.flags.DEFINE_integer('batch_size', 1, 'Batch size to use')
-tf_v1.app.flags.DEFINE_string('data_dir', '', 'Path to images in TF records format')
-FLAGS = tf_v1.app.flags.FLAGS
-MODEL = 'resnet50v1_5'
-IMAGE_SIZE = 224
-
-
-def preprocess_image(image_buffer, model, image_size):
-    """Preprocess JPEG encoded bytes to 3D float Tensor."""
-
-    # Decode the string as an RGB JPEG of unknown height and width.
-    image = tf_v1.image.decode_jpeg(image_buffer, channels=3)
-    # Convert pixels to [0, 1)
-    image = tf_v1.image.convert_image_dtype(image, dtype=tf.float32)
-    # Crop the central region to 87.5% of the original image.
-    image = tf_v1.image.central_crop(image, central_fraction=0.875)
-    # Resize the image to image_size x image_size.
-    image = tf_v1.expand_dims(image, 0)
-    image = tf_v1.image.resize_bilinear(image, [image_size, image_size], align_corners=False)
-    image = tf_v1.squeeze(image, [0])
-    # For ResNet50, rescale to [0, 256]
-    image = tf_v1.multiply(image, 256.0)
-
-    return image
-
-
-def parse_example_proto(example_serialized):
-    # Dense features in Example proto.
-    feature_map = {
-        'image/encoded': tf_v1.FixedLenFeature([], dtype=tf_v1.string,
-                                            default_value=''),
-        'image/class/label': tf_v1.FixedLenFeature([1], dtype=tf_v1.int64,
-                                                default_value=-1),
-    }
-
-    features = tf_v1.parse_single_example(example_serialized, feature_map)
-    label = tf_v1.cast(features['image/class/label'], dtype=tf.int32)
-
-    return features['image/encoded'], label
-
-
-def sample_images(image_size):
-    """Pull a random batch of images from FLAGS.data_dir containing TF record formatted ImageNet validation set
-    Returns:
-        ndarray of float32 with shape [FLAGS.batch_size, image_size, image_size, 3]
-    """
-
-    sample_file = random.choice(os.listdir(FLAGS.data_dir))
-    dataset = tf.data.TFRecordDataset(os.path.join(FLAGS.data_dir, sample_file))
-    dataset = dataset.map(lambda x: parse_example_proto(x)).shuffle(True).batch(FLAGS.batch_size)
-    iterator = dataset.make_one_shot_iterator()
-    next_element = iterator.get_next()
-    with tf.Session() as sess:
-        images, labels = sess.run(next_element)
-        images = np.array([sess.run(preprocess_image(x, MODEL, image_size)) for x in images])
-
-    return images
-
-
-def main(_):
-    channel = grpc.insecure_channel(FLAGS.server)
-    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
-    i = 0
-    num_iteration = 40
-    warm_up_iteration = 10
-    total_time = 0
-    for _ in range(num_iteration):
-        i += 1
-        if FLAGS.data_dir:
-            image_np = sample_images(IMAGE_SIZE)
-        else:
-            image_np = np.random.rand(FLAGS.batch_size, IMAGE_SIZE, IMAGE_SIZE, 3).astype(np.float32)
-            image_np *= 256.0
-
-        request = predict_pb2.PredictRequest()
-        request.model_spec.name = MODEL
-        request.model_spec.signature_name = 'serving_default'
-        request.inputs['input_tensor'].CopyFrom(
-            tf.make_tensor_proto(image_np, shape=[FLAGS.batch_size, IMAGE_SIZE, IMAGE_SIZE, 3]))
-        start_time = time.time()
-        stub.Predict(request, 10.0)  # 10 secs timeout
-        time_consume = time.time() - start_time
-        print('Iteration %d: %.3f sec' % (i, time_consume))
-        if i > warm_up_iteration:
-            total_time += time_consume
-
-    time_average = total_time / (num_iteration - warm_up_iteration)
-    print('Average time: %.3f sec' % (time_average))
-
-    print('Batch size = %d' % FLAGS.batch_size)
-    if (FLAGS.batch_size == 1):
-        print('Latency: %.3f ms' % (time_average * 1000))
-
-    print('Throughput: %.3f images/sec' % (FLAGS.batch_size / time_average))
-
-
-if __name__ == '__main__':
-    tf_v1.app.run()
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/datasets.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/datasets.md
deleted file mode 100644
index 9d4ca6244..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-The ImageNet dataset is used in these <model name> Kubernetes examples.
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/description.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/description.md
deleted file mode 100644
index 07a043dac..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/description.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/download.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/kubernetes.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/kubernetes.md
deleted file mode 100644
index 34f0d7869..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/kubernetes.md
+++ /dev/null
@@ -1,249 +0,0 @@
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the ResNet50 v1.5 FP32 training package:
-```
-wget <package url>
-tar -xvf <package name>
-```
-
-#### Prerequisites
-
-Both single and multi-node deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-### Execution
-
-The Kubernetes* package for `<model name> <precision> <mode>` includes single and multi-node kubernetes deployments.
-Within the single and multi-node deployments are common use cases that include storage and security 
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below, 
-where single and multi-node directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory. 
-Common use cases are found under the single and multi-node directories:
-
-```
-quickstart
-└── mlops
-      ├── multi-node
-      │       ├── user-allocated-pvc
-      │       └── user-mounted-nfs
-      └── single-node
-              ├── user-allocated-pvc
-              └── user-mounted-nfs
-```
-
-#### Multi-node distributed training
-
-The multi-node use cases (user-allocated-pvc, user-mounted-nfs) make the following assumptions:
-- the [mpi-operator](/tools/k8s/devops/operators/mpi-operator.yaml) has been deployed on the cluster by [devops](https://en.wikipedia.org/wiki/DevOps) (see below).
-- the OUTPUT_DIR parameter is a shared volume that is writable by the user and available cluster wide.
-- the DATASET_DIR parameter is a dataset volume also available cluster wide (eg: using zfs or other performant storage). Typically these volumes are read-only.
-
-##### [Devops](https://en.wikipedia.org/wiki/DevOps)
-
-The k8 resources needed to run the multi-node <package dir> quickstart require deployment of an mpi-operator.
-See the [MPI Operator deployment](/k8s/common/tensorflow/KubernetesDevOps.md#mpi-operator-deployment) section of the Kubernetes DevOps document for instructions.
-
-Once these resources have been deployed, the mlops user then has a choice 
-of running <package dir> multi-node (distributed training) or single-node. 
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-Distributed training is done by posting an MPIJob to the k8s api-server which is handled by the mpi-operator that was deployed by 
-devops. The mpi-operator parses the MPIJob and then runs a launcher and workers specified in the MPIJob. Launcher and workers communicate through [horovod](https://github.com/horovod/horovod).
-The distributed training algorithm is handled by [mpirun](https://www.open-mpi.org/doc/current/man1/mpirun.1.php). 
-
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | pvc name                    |
-| PVC_PATH   | /pvc                     | pvc path                    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Multi-node training output
-
-Viewing the log output of the resnet50v1_5 MPIJob is done by viewing the logs of the 
-launcher pod. The launcher pod aggregrates output from the workerpods. 
-This pod is found by filtering the list of pods for the name 'launcher'
-
-```
-kubectl get pods -oname|grep launch|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep launch|cut -c5-)
-```
-
-Note that the mpirun parameter -output-filename is actually a directory and is set to $OUTPUT_DIR.
-
-##### Multi-node training cleanup
-
-Removing the mpijob and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Single-node training
-
-Single node training is similar to the docker use case but the command is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running 
-the fp32_training_demo.sh command within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the resnet50v1_5 Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/license.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/quickstart.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/quickstart.md
deleted file mode 100644
index d04b209ed..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L18) | Executes a short run using small batch sizes and a limited number of steps to demonstrate the training flow |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/title.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/troubleshooting.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/troubleshooting.md
deleted file mode 100644
index fd80903fb..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.docs/troubleshooting.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/deployments-yml-exist.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/deployments-yml-exist.sh
deleted file mode 100644
index d30d886cc..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/deployments-yml-exist.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-load "lib/test_helper"
-
-@test "validate package for resnet50v1-5-fp32-training-k8s in framework k8s creates model package file" {
-  run model-builder package -f k8s resnet50v1-5-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f output/resnet50v1-5-fp32-training-k8s.tar.gz ]]
-}
-
-@test "validate generate-deployment for resnet50v1-5-fp32-training-k8s in framework k8s creates deployment files under deployments/resnet50v1-5-fp32-training-k8s/" {
-  run model-builder generate-deployment -f k8s resnet50v1-5-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f deployments/resnet50v1-5-fp32-training-k8s/multi-node/user-allocated-pvc.yaml ]]
-  [[ -f deployments/resnet50v1-5-fp32-training-k8s/multi-node/user-mounted-nfs.yaml ]]
-  [[ -f deployments/resnet50v1-5-fp32-training-k8s/single-node/user-allocated-pvc.yaml ]]
-  [[ -f deployments/resnet50v1-5-fp32-training-k8s/single-node/user-mounted-nfs.yaml ]]
-  (( $(last_modified deployments/resnet50v1-5-fp32-training-k8s/multi-node/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/resnet50v1-5-fp32-training-k8s/multi-node/user-mounted-nfs.yaml) <= 50 ))
-  (( $(last_modified deployments/resnet50v1-5-fp32-training-k8s/single-node/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/resnet50v1-5-fp32-training-k8s/single-node/user-mounted-nfs.yaml) <= 50 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/namespace-exists.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/namespace-exists.sh
deleted file mode 100644
index 4a76a6981..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/namespace-exists.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "verify namespace $USER_NAME exists" {
-  run verify "there is 1 namespace named '"$USER_NAME"'"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/single_node_deployment.sh b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/single_node_deployment.sh
deleted file mode 100644
index 96309dab4..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/.tests/single_node_deployment.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "kubectl apply of single-node resnet50v1-5-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME apply -f deployments/resnet50v1-5-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "single-node deployment of resnet50v1-5-fp32-training-k8s creates 1 pod and it is running" {
-  try "at most 4 times every 30s to get pods named 'resnet50v1-5-fp32-training' and verify that 'status' is 'running'"
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "verify job completion, check end of output includes text 'run_final'" {
-  run launcher_logs
-  echo "output = ${output}"
-  (( $status == 0 ))
-  [[ ${lines[4]} =~ run_final ]]
-}
-
-@test "kubectl delete of single-node resnet50v1-5-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME delete -f deployments/resnet50v1-5-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/README.md b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/README.md
deleted file mode 100644
index 0ca1192fb..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/README.md
+++ /dev/null
@@ -1,306 +0,0 @@
-<!--- 0. Title -->
-# ResNet50 v1.5 FP32 training
-
-<!-- 10. Description -->
-
-This document has instructions for running ResNet50 v1.5 FP32 training using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
-
-<!--- 20. Download link -->
-## Download link
-
-[resnet50v1-5-fp32-training-k8s.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/resnet50v1-5-fp32-training-k8s.tar.gz)
-
-<!--- 30. Datasets -->
-## Dataset
-
-The ImageNet dataset is used in these ResNet50 v1.5 Kubernetes examples.
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L18) | Executes a short run using small batch sizes and a limited number of steps to demonstrate the training flow |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
-
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the ResNet50 v1.5 FP32 training package:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/resnet50v1-5-fp32-training-k8s.tar.gz
-tar -xvf resnet50v1-5-fp32-training-k8s.tar.gz
-```
-
-#### Prerequisites
-
-Both single and multi-node deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-### Execution
-
-The Kubernetes* package for `ResNet50 v1.5 FP32 training` includes single and multi-node kubernetes deployments.
-Within the single and multi-node deployments are common use cases that include storage and security 
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below, 
-where single and multi-node directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory. 
-Common use cases are found under the single and multi-node directories:
-
-```
-quickstart
-└── mlops
-      ├── multi-node
-      │       ├── user-allocated-pvc
-      │       └── user-mounted-nfs
-      └── single-node
-              ├── user-allocated-pvc
-              └── user-mounted-nfs
-```
-
-#### Multi-node distributed training
-
-The multi-node use cases (user-allocated-pvc, user-mounted-nfs) make the following assumptions:
-- the [mpi-operator](/tools/k8s/devops/operators/mpi-operator.yaml) has been deployed on the cluster by [devops](https://en.wikipedia.org/wiki/DevOps) (see below).
-- the OUTPUT_DIR parameter is a shared volume that is writable by the user and available cluster wide.
-- the DATASET_DIR parameter is a dataset volume also available cluster wide (eg: using zfs or other performant storage). Typically these volumes are read-only.
-
-##### [Devops](https://en.wikipedia.org/wiki/DevOps)
-
-The k8 resources needed to run the multi-node resnet50v1-5-fp32-training-k8s quickstart require deployment of an mpi-operator.
-See the [MPI Operator deployment](/k8s/common/tensorflow/KubernetesDevOps.md#mpi-operator-deployment) section of the Kubernetes DevOps document for instructions.
-
-Once these resources have been deployed, the mlops user then has a choice 
-of running resnet50v1-5-fp32-training-k8s multi-node (distributed training) or single-node. 
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-Distributed training is done by posting an MPIJob to the k8s api-server which is handled by the mpi-operator that was deployed by 
-devops. The mpi-operator parses the MPIJob and then runs a launcher and workers specified in the MPIJob. Launcher and workers communicate through [horovod](https://github.com/horovod/horovod).
-The distributed training algorithm is handled by [mpirun](https://www.open-mpi.org/doc/current/man1/mpirun.1.php). 
-
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | pvc name                    |
-| PVC_PATH   | /pvc                     | pvc path                    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Multi-node training output
-
-Viewing the log output of the resnet50v1_5 MPIJob is done by viewing the logs of the 
-launcher pod. The launcher pod aggregrates output from the workerpods. 
-This pod is found by filtering the list of pods for the name 'launcher'
-
-```
-kubectl get pods -oname|grep launch|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep launch|cut -c5-)
-```
-
-Note that the mpirun parameter -output-filename is actually a directory and is set to $OUTPUT_DIR.
-
-##### Multi-node training cleanup
-
-Removing the mpijob and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Single-node training
-
-Single node training is similar to the docker use case but the command is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running 
-the fp32_training_demo.sh command within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the resnet50v1_5 Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile
deleted file mode 100644
index c4939a409..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,205 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      description: owner id of mounted volumes
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.FS_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: FS_ID_VALUE
-          pattern: "${FS_ID}"
-          values:
-          - marker: ${FS_ID}
-            ref: '#/definitions/io.k8s.cli.setters.FS_ID'
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/image-recognition:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/resnet50v1-5-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: resnet50v1-5-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_SCRIPT:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_SCRIPT
-          pattern: ${MODEL_DIR}/models/image_recognition/tensorflow/resnet50v1_5/training/mlperf_resnet/imagenet_main.py
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.STORAGE:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: STORAGE
-          value: 1Mi
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.WORKERS:
-      description: number of workers
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: WORKERS
-          value: "2"
-          isSet: true
-          setBy: model-builder
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index f66235b54..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/resnet50v1-5-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index f1f32163c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- service-account.yaml
-- config-map.yaml
-- mpi-job.yaml
-namespace: default
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml
deleted file mode 100644
index c78ae080d..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml
+++ /dev/null
@@ -1,113 +0,0 @@
-apiVersion: kubeflow.org/v1alpha2
-kind: MPIJob
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  slotsPerWorker: 1
-  cleanPodPolicy: Running
-  mpiReplicaSpecs:
-    Launcher:
-      replicas: 1
-      template:
-        spec:
-          serviceAccountName: resnet50v1-5-fp32-training
-          containers:
-          - name: mpi-launcher
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-            command:
-            - mpirun
-            args:
-            - --allow-run-as-root
-            - -np
-            - "2"
-            - -x
-            - PYTHONPATH
-            - -x
-            - LD_LIBRARY_PATH
-            - -mca
-            - pml
-            - ob1
-            - -mca
-            - btl
-            - ^openib
-            - -mca
-            - btl_tcp_if_exclude
-            - lo,docker0
-            - -bind-to
-            - none
-            - -map-by
-            - slot
-            - -output-filename
-            - $(OUTPUT_DIR)
-            - python3
-            - /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training/mlperf_resnet/imagenet_main.py # {"$openapi":"MODEL_SCRIPT"}
-            - "2"
-            - --batch_size
-            - "128"
-            - --max_train_steps
-            - "5"
-            - --train_epochs
-            - "1"
-            - --epochs_between_evals
-            - "1"
-            - --inter_op_parallelism_threads
-            - "2"
-            - --intra_op_parallelism_threads
-            - "22"
-            - --version
-            - "1"
-            - --resnet_size
-            - "50"
-            - --data_dir=$(DATASET_DIR)
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: pvc-path
-              mountPath: /pvc # {"$openapi":"PVC_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: pvc-path
-            persistentVolumeClaim:
-              claimName: workdisk # {"$openapi":"PVC_NAME"}
-    Worker:
-      replicas: 2 # {"$openapi":"WORKERS"}
-      template:
-        spec:
-          serviceAccountName: resnet50v1-5-fp32-training
-          containers:
-          - name: mpi-worker
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: pvc-path
-              mountPath: /pvc # {"$openapi":"PVC_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: pvc-path
-            persistentVolumeClaim:
-              claimName: workdisk # {"$openapi":"PVC_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 5d7123480..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile
deleted file mode 100644
index 5ee9f8542..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,196 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      description: owner id of mounted volumes
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.FS_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: FS_ID_VALUE
-          pattern: "${FS_ID}"
-          values:
-          - marker: ${FS_ID}
-            ref: '#/definitions/io.k8s.cli.setters.FS_ID'
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/image-recognition:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/resnet50v1-5-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: resnet50v1-5-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_SCRIPT:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_SCRIPT
-          pattern: ${MODEL_DIR}/models/image_recognition/tensorflow/resnet50v1_5/training/mlperf_resnet/imagenet_main.py
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.WORKERS:
-      description: number of workers
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: WORKERS
-          value: "2"
-          isSet: true
-          setBy: model-builder
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 50cbcafa1..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/resnet50v1-5-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index f1f32163c..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- service-account.yaml
-- config-map.yaml
-- mpi-job.yaml
-namespace: default
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml
deleted file mode 100644
index ee5489af0..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml
+++ /dev/null
@@ -1,116 +0,0 @@
-apiVersion: kubeflow.org/v1alpha2
-kind: MPIJob
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  slotsPerWorker: 1
-  MPIDistribution: IntelMPI
-  cleanPodPolicy: Running
-  mpiReplicaSpecs:
-    Launcher:
-      replicas: 1
-      template:
-        spec:
-          serviceAccountName: resnet50v1-5-fp32-training
-          containers:
-          - name: mpi-launcher
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-            command:
-            - mpirun
-            args:
-            - --allow-run-as-root
-            - -np
-            - "2"
-            - -x
-            - PYTHONPATH
-            - -x
-            - LD_LIBRARY_PATH
-            - -mca
-            - pml
-            - ob1
-            - -mca
-            - btl
-            - ^openib
-            - -mca
-            - btl_tcp_if_exclude
-            - lo,docker0
-            - -bind-to
-            - none
-            - -map-by
-            - slot
-            - -output-filename
-            - $(OUTPUT_DIR)
-            - python3
-            - /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training/mlperf_resnet/imagenet_main.py # {"$openapi":"MODEL_SCRIPT"}
-            - "2"
-            - --batch_size
-            - "128"
-            - --max_train_steps
-            - "5"
-            - --train_epochs
-            - "1"
-            - --epochs_between_evals
-            - "1"
-            - --inter_op_parallelism_threads
-            - "2"
-            - --intra_op_parallelism_threads
-            - "22"
-            - --version
-            - "1"
-            - --resnet_size
-            - "50"
-            - --data_dir=$(DATASET_DIR)
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: nfs-path
-              mountPath: /nfs # {"$openapi":"NFS_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: nfs-path
-            nfs:
-              server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-              path: /nfs # {"$openapi":"NFS_PATH"}
-    Worker:
-      replicas: 2 # {"$openapi":"WORKERS"}
-      template:
-        spec:
-          serviceAccountName: resnet50v1-5-fp32-training
-          containers:
-          - name: mpi-worker
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: nfs-path
-              mountPath: /nfs # {"$openapi":"NFS_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: nfs-path
-            nfs:
-              server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-              path: /nfs # {"$openapi":"NFS_PATH"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 5d7123480..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
deleted file mode 100644
index 8502dc111..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,206 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: resnet50v1-5-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      description: owner id of mounted volumes
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.FS_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: FS_ID_VALUE
-          pattern: "${FS_ID}"
-          values:
-          - marker: ${FS_ID}
-            ref: '#/definitions/io.k8s.cli.setters.FS_ID'
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/image-recognition:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/resnet50v1-5-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.WORKERS:
-      description: number of workers
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: WORKERS
-          value: "2"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: launch_benchmark.py
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/benchmarks/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index f66235b54..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/resnet50v1-5-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index 251cd64b1..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- service-account.yaml
-- config-map.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
deleted file mode 100644
index 8a40584b7..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
+++ /dev/null
@@ -1,50 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-    workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/resnet50v1-5-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=resnet50v1_5
-    - --precision=fp32
-    - --mode=training
-    - --framework
-    - tensorflow
-    - --batch-size
-    - "16"
-    - --checkpoint
-    - $(OUTPUT_DIR)
-    - --data-location=$(DATASET_DIR)
-    - --noinstall
-    - --
-    - steps=50
-    - train_epochs=1
-    - epochs_between_evals=1
-    envFrom:
-    - configMapRef:
-        name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: pvc-path
-    persistentVolumeClaim:
-      claimName: workdisk # {"$openapi":"PVC_NAME"}
-  restartPolicy: OnFailure
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 5d7123480..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
deleted file mode 100644
index 32752302f..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,189 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: resnet50v1-5-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/image-recognition:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/resnet50v1-5-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: launch_benchmark.py
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/benchmarks/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 50cbcafa1..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/resnet50v1-5-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/resnet50v1-5-fp32-training/models/image_recognition/tensorflow/resnet50v1_5/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index 251cd64b1..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- service-account.yaml
-- config-map.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
deleted file mode 100644
index 870ca77e2..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
+++ /dev/null
@@ -1,51 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/image-recognition:tf-latest-resnet50v1-5-fp32-training # {"$openapi":"IMAGE"}
-    workingDir: /workspace/resnet50v1-5-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/resnet50v1-5-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=resnet50v1_5
-    - --precision=fp32
-    - --mode=training
-    - --framework
-    - tensorflow
-    - --batch-size
-    - "16"
-    - --checkpoint
-    - $(OUTPUT_DIR)
-    - --data-location=$(DATASET_DIR)
-    - --noinstall
-    - --
-    - steps=50
-    - train_epochs=1
-    - epochs_between_evals=1
-    envFrom:
-    - configMapRef:
-        name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: nfs-path
-    nfs:
-      server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-      path: /nfs # {"$openapi":"NFS_PATH"}
-  restartPolicy: OnFailure
diff --git a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml b/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 5d7123480..000000000
--- a/k8s/image_recognition/tensorflow/resnet50v1_5/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: resnet50v1-5-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/datasets.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/datasets.md
deleted file mode 100644
index 4361dbea1..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,26 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-### Pretrained models
-
-Download and extract checkpoints the bert pretrained model from the
-[google bert repo](https://github.com/google-research/bert#pre-trained-models).
-The extracted directory should be set to the `CHECKPOINT_DIR` environment
-variable when running the quickstart scripts.
-
-For training from scratch, Wikipedia and BookCorpus need to be downloaded
-and pre-processed.
-
-### GLUE data
-
-[GLUE data](https://gluebenchmark.com/tasks) is used when running BERT
-classification training. Download and unpack the GLUE data by running
-[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e).
-
-### SQuAD data
-
-The Stanford Question Answering Dataset (SQuAD) dataset files can be downloaded
-from the [Google bert repo](https://github.com/google-research/bert#squad-11).
-The three files (`train-v1.1.json`, `dev-v1.1.json`, and `evaluate-v1.1.py`)
-should be downloaded to the same directory. Set the `DATASET_DIR` to point to
-that directory when running bert fine tuning using the SQuAD data.
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/description.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/description.md
deleted file mode 100644
index 0b83973a5..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/description.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running [BERT](https://github.com/google-research/bert#what-is-bert) <precision> <mode> using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-For all fine-tuning the datasets (SQuAD, MultiNLI, MRPC etc..) and checkpoints
-should be downloaded as mentioned in the [Google bert repo](https://github.com/google-research/bert).
-
-Refer to google reference page for [checkpoints](https://github.com/google-research/bert#pre-trained-models).
-
-
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/download.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/kubernetes.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/kubernetes.md
deleted file mode 100644
index 6a9bae8fb..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/kubernetes.md
+++ /dev/null
@@ -1,247 +0,0 @@
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the <model name> <precision> <mode> package:
-```
-wget <package url>
-tar -xvf <package name>
-```
-
-#### Prerequisites
-
-Both single and multi-node deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-### Execution
-
-The Kubernetes* package for `<model name> <precision> <mode>` includes single and multi-node kubernetes deployments.
-Within the single and multi-node deployments are common use cases that include storage and security
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below,
-where single and multi-node directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory.
-Common use cases are found under the single and multi-node directories:
-
-```
-quickstart
-└── mlops
-      ├── multi-node
-      │       ├── user-allocated-pvc
-      │       └── user-mounted-nfs
-      └── single-node
-              ├── user-allocated-pvc
-              └── user-mounted-nfs
-```
-
-#### Multi-node distributed training
-
-The multi-node use cases (user-allocated-pvc, user-mounted-nfs) make the following assumptions:
-- the [mpi-operator](/tools/k8s/devops/operators/mpi-operator.yaml) has been deployed on the cluster by [devops](https://en.wikipedia.org/wiki/DevOps) (see below).
-- the OUTPUT_DIR parameter is a shared volume that is writable by the user and available cluster wide.
-- the DATASET_DIR parameter is a dataset volume also available cluster wide (eg: using zfs or other performant storage). Typically these volumes are read-only.
-
-##### [Devops](https://en.wikipedia.org/wiki/DevOps)
-
-The k8 resources needed to run the multi-node <package dir> quickstart require deployment of an mpi-operator.
-See the [MPI Operator deployment](/k8s/common/tensorflow/KubernetesDevOps.md#mpi-operator-deployment) section of the Kubernetes DevOps document for instructions.
-
-Once these resources have been deployed, the mlops user then has a choice 
-of running <package dir> multi-node (distributed training) or single-node. 
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-Distributed training is done by posting an MPIJob to the k8s api-server which is handled by the mpi-operator that was deployed by 
-devops. The mpi-operator parses the MPIJob and then runs a launcher and workers specified in the MPIJob. Launcher and workers communicate through [horovod](https://github.com/horovod/horovod).
-The distributed training algorithm is handled by [mpirun](https://www.open-mpi.org/doc/current/man1/mpirun.1.php). 
-
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Multi-node training output
-
-Viewing the log output of the <package name> MPIJob is done by viewing the logs of the 
-launcher pod. The launcher pod aggregrates output from the workerpods. 
-This pod is found by filtering the list of pods for the name 'launcher'
-
-```
-kubectl get pods -oname|grep launch|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep launch|cut -c5-)
-```
-
-Note that the mpirun parameter -output-filename is actually a directory and is set to $OUTPUT_DIR.
-
-##### Multi-node training cleanup
-
-Removing the mpijob and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Single-node training
-
-Single node training is similar to the docker use case but the command is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running 
-the fp32_training_demo.sh command within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the <package name> Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/license.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/quickstart.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/quickstart.md
deleted file mode 100644
index b670333c3..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L18) | This script is used by the single node Kubernetes job to run bert classifier inference. |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/title.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/troubleshooting.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/troubleshooting.md
deleted file mode 100644
index fd80903fb..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.docs/troubleshooting.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/deployments-yml-exist.sh b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/deployments-yml-exist.sh
deleted file mode 100644
index 58f98ee22..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/deployments-yml-exist.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-load "lib/test_helper"
-
-@test "validate package for bert-large-fp32-training-k8s in framework k8s creates model package file" {
-  run model-builder package -f k8s bert-large-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f output/bert-large-fp32-training-k8s.tar.gz ]]
-}
-
-@test "validate generate-deployment for bert-large-fp32-training-k8s in framework k8s creates deployment files under deployments/bert-large-fp32-training-k8s/" {
-  run model-builder generate-deployment -f k8s bert-large-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f deployments/bert-large-fp32-training-k8s/multi-node/user-allocated-pvc.yaml ]]
-  [[ -f deployments/bert-large-fp32-training-k8s/multi-node/user-mounted-nfs.yaml ]]
-  [[ -f deployments/bert-large-fp32-training-k8s/single-node/user-allocated-pvc.yaml ]]
-  [[ -f deployments/bert-large-fp32-training-k8s/single-node/user-mounted-nfs.yaml ]]
-  (( $(last_modified deployments/bert-large-fp32-training-k8s/multi-node/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/bert-large-fp32-training-k8s/multi-node/user-mounted-nfs.yaml) <= 50 ))
-  (( $(last_modified deployments/bert-large-fp32-training-k8s/single-node/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/bert-large-fp32-training-k8s/single-node/user-mounted-nfs.yaml) <= 50 ))
-}
\ No newline at end of file
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/namespace-exists.sh b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/namespace-exists.sh
deleted file mode 100644
index 4a76a6981..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/namespace-exists.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "verify namespace $USER_NAME exists" {
-  run verify "there is 1 namespace named '"$USER_NAME"'"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/single_node_deployment.sh b/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/single_node_deployment.sh
deleted file mode 100644
index 49188da05..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/single_node_deployment.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "kubectl apply of single-node bert-large-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME apply -f deployments/bert-large-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "single-node deployment of bert-large-fp32-training-k8s creates 1 pod and it is running" {
-  try "at most 4 times every 30s to get pods named 'bert-large-fp32-training' and verify that 'status' is 'running'"
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "verify job completion, check end of output includes text 'run_final'" {
-  run launcher_logs
-  echo "output = ${output}"
-  (( $status == 0 ))
-  [[ ${lines[4]} =~ run_final ]]
-}
-
-@test "kubectl delete of single-node bert-large-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME delete -f deployments/bert-large-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/README.md b/k8s/language_modeling/tensorflow/bert_large/training/fp32/README.md
deleted file mode 100644
index 3af52ae7a..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/README.md
+++ /dev/null
@@ -1,328 +0,0 @@
-<!--- 0. Title -->
-# BERT Large FP32 training
-
-<!-- 10. Description -->
-
-This document has instructions for running [BERT](https://github.com/google-research/bert#what-is-bert) FP32 training using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-For all fine-tuning the datasets (SQuAD, MultiNLI, MRPC etc..) and checkpoints
-should be downloaded as mentioned in the [Google bert repo](https://github.com/google-research/bert).
-
-Refer to google reference page for [checkpoints](https://github.com/google-research/bert#pre-trained-models).
-
-
-
-<!--- 20. Download link -->
-## Download link
-
-[bert-large-fp32-training-k8s.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/bert-large-fp32-training-k8s.tar.gz)
-
-<!--- 30. Datasets -->
-## Datasets
-
-### Pretrained models
-
-Download and extract checkpoints the bert pretrained model from the
-[google bert repo](https://github.com/google-research/bert#pre-trained-models).
-The extracted directory should be set to the `CHECKPOINT_DIR` environment
-variable when running the quickstart scripts.
-
-For training from scratch, Wikipedia and BookCorpus need to be downloaded
-and pre-processed.
-
-### GLUE data
-
-[GLUE data](https://gluebenchmark.com/tasks) is used when running BERT
-classification training. Download and unpack the GLUE data by running
-[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e).
-
-### SQuAD data
-
-The Stanford Question Answering Dataset (SQuAD) dataset files can be downloaded
-from the [Google bert repo](https://github.com/google-research/bert#squad-11).
-The three files (`train-v1.1.json`, `dev-v1.1.json`, and `evaluate-v1.1.py`)
-should be downloaded to the same directory. Set the `DATASET_DIR` to point to
-that directory when running bert fine tuning using the SQuAD data.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L18) | This script is used by the single node Kubernetes job to run bert classifier inference. |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
-
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the BERT Large FP32 training package:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/bert-large-fp32-training-k8s.tar.gz
-tar -xvf bert-large-fp32-training-k8s.tar.gz
-```
-
-#### Prerequisites
-
-Both single and multi-node deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-### Execution
-
-The Kubernetes* package for `BERT Large FP32 training` includes single and multi-node kubernetes deployments.
-Within the single and multi-node deployments are common use cases that include storage and security
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below,
-where single and multi-node directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory.
-Common use cases are found under the single and multi-node directories:
-
-```
-quickstart
-└── mlops
-      ├── multi-node
-      │       ├── user-allocated-pvc
-      │       └── user-mounted-nfs
-      └── single-node
-              ├── user-allocated-pvc
-              └── user-mounted-nfs
-```
-
-#### Multi-node distributed training
-
-The multi-node use cases (user-allocated-pvc, user-mounted-nfs) make the following assumptions:
-- the [mpi-operator](/tools/k8s/devops/operators/mpi-operator.yaml) has been deployed on the cluster by [devops](https://en.wikipedia.org/wiki/DevOps) (see below).
-- the OUTPUT_DIR parameter is a shared volume that is writable by the user and available cluster wide.
-- the DATASET_DIR parameter is a dataset volume also available cluster wide (eg: using zfs or other performant storage). Typically these volumes are read-only.
-
-##### [Devops](https://en.wikipedia.org/wiki/DevOps)
-
-The k8 resources needed to run the multi-node bert-large-fp32-training-k8s quickstart require deployment of an mpi-operator.
-See the [MPI Operator deployment](/k8s/common/tensorflow/KubernetesDevOps.md#mpi-operator-deployment) section of the Kubernetes DevOps document for instructions.
-
-Once these resources have been deployed, the mlops user then has a choice 
-of running bert-large-fp32-training-k8s multi-node (distributed training) or single-node. 
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-Distributed training is done by posting an MPIJob to the k8s api-server which is handled by the mpi-operator that was deployed by 
-devops. The mpi-operator parses the MPIJob and then runs a launcher and workers specified in the MPIJob. Launcher and workers communicate through [horovod](https://github.com/horovod/horovod).
-The distributed training algorithm is handled by [mpirun](https://www.open-mpi.org/doc/current/man1/mpirun.1.php). 
-
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Multi-node training output
-
-Viewing the log output of the bert-large-fp32-training-k8s.tar.gz MPIJob is done by viewing the logs of the 
-launcher pod. The launcher pod aggregrates output from the workerpods. 
-This pod is found by filtering the list of pods for the name 'launcher'
-
-```
-kubectl get pods -oname|grep launch|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep launch|cut -c5-)
-```
-
-Note that the mpirun parameter -output-filename is actually a directory and is set to $OUTPUT_DIR.
-
-##### Multi-node training cleanup
-
-Removing the mpijob and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Single-node training
-
-Single node training is similar to the docker use case but the command is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running 
-the fp32_training_demo.sh command within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the bert-large-fp32-training-k8s.tar.gz Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile
deleted file mode 100644
index 2aa9fb07c..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,241 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: bert-large-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.GLUE_DIR:
-      description: glue dir
-      x-k8s-cli:
-        setter:
-          name: GLUE_DIR
-          value: bert_official
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.BERT_BASE_DIR:
-      description: bert base dir
-      x-k8s-cli:
-        setter:
-          name: BERT_BASE_DIR
-          value: bert_official/MRPC/uncased_L-12_H-768_A-12
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint directory
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: /checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      description: owner id of mounted volumes
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/language-modeling:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/bert-large-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_SCRIPT:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_SCRIPT
-          pattern: ${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.WORKERS:
-      description: number of workers
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: WORKERS
-          value: "2"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.PYTHONPATH:
-      x-k8s-cli:
-        substitution:
-          name: PYTHONPATH
-          pattern: ${MODEL_DIR}/benchmarks:${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index 25a9cc8d1..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  BERT_BASE_DIR: bert_official/MRPC/uncased_L-12_H-768_A-12 # {"$openapi":"BERT_BASE_DIR"}
-  CHECKPOINT_DIR: /checkpoints # {"$openapi":"CHECKPOINT_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GLUE_DIR: bert_official # {"$openapi":"GLUE_DIR"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/bert-large-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/bert-large-fp32-training/benchmarks:/workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index 3ec410507..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- mpi-job.yaml
-namespace: default
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml
deleted file mode 100644
index 759dd6cc0..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/mpi-job.yaml
+++ /dev/null
@@ -1,110 +0,0 @@
-apiVersion: kubeflow.org/v1alpha2
-kind: MPIJob
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  slotsPerWorker: 1
-  cleanPodPolicy: Running
-  mpiReplicaSpecs:
-    Launcher:
-      replicas: 1
-      template:
-        spec:
-          serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-          containers:
-          - name: mpi-launcher
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-            command:
-            - mpirun
-            args:
-            - --allow-run-as-root
-            - -np
-            - "2"
-            - -x
-            - PYTHONPATH
-            - -x
-            - LD_LIBRARY_PATH
-            - -mca
-            - pml
-            - ob1
-            - -mca
-            - btl
-            - ^openib
-            - -mca
-            - btl_tcp_if_exclude
-            - lo,docker0
-            - -bind-to
-            - none
-            - -map-by
-            - slot
-            - -output-filename
-            - $(OUTPUT_DIR)
-            - python3
-            - /workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py # {"$openapi":"MODEL_SCRIPT"}
-            - --task_name=MRPC
-            - --do_train=true
-            - --do_eval=true
-            - --data_dir=$(DATASET_DIR)/$(GLUE_DIR)/MRPC
-            - --vocab_file=$(DATASET_DIR)/$(BERT_BASE_DIR)/vocab.txt
-            - --bert_config_file=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_config.json
-            - --init_checkpoint=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_model.ckpt
-            - --max_seq_length=128
-            - --train_batch_size=32
-            - --learning_rate=2e-5
-            - --num_train_epochs=3.0
-            - --output_dir=$(OUTPUT_DIR)
-            - --use_tpu=False
-            - --precision=fp32
-            - --do_lower_case=True
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: pvc-path
-              mountPath: /pvc # {"$openapi":"PVC_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: pvc-path
-            persistentVolumeClaim:
-              claimName: workdisk # {"$openapi":"PVC_NAME"}
-    Worker:
-      replicas: 2 # {"$openapi":"WORKERS"}
-      template:
-        spec:
-          serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-          containers:
-          - name: mpi-worker
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: pvc-path
-              mountPath: /pvc # {"$openapi":"PVC_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: pvc-path
-            persistentVolumeClaim:
-              claimName: workdisk # {"$openapi":"PVC_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index b26e65897..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile
deleted file mode 100644
index 4b5b963f1..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,223 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: bert-large-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.GLUE_DIR:
-      description: glue dir
-      x-k8s-cli:
-        setter:
-          name: GLUE_DIR
-          value: bert_official
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.BERT_BASE_DIR:
-      description: bert base dir
-      x-k8s-cli:
-        setter:
-          name: BERT_BASE_DIR
-          value: bert_official/MRPC/uncased_L-12_H-768_A-12
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint directory
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: /checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      description: owner id of mounted volumes
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/language-modeling:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/bert-large-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_SCRIPT:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_SCRIPT
-          pattern: ${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.WORKERS:
-      description: number of workers
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: WORKERS
-          value: "2"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.substitutions.PYTHONPATH:
-      x-k8s-cli:
-        substitution:
-          name: PYTHONPATH
-          pattern: ${MODEL_DIR}/benchmarks:${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 70c917e62..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  BERT_BASE_DIR: bert_official/MRPC/uncased_L-12_H-768_A-12 # {"$openapi":"BERT_BASE_DIR"}
-  CHECKPOINT_DIR: /checkpoints # {"$openapi":"CHECKPOINT_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GLUE_DIR: bert_official # {"$openapi":"GLUE_DIR"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/bert-large-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/bert-large-fp32-training/benchmarks:/workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index 3ec410507..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- mpi-job.yaml
-namespace: default
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml
deleted file mode 100644
index 59a58042b..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/mpi-job.yaml
+++ /dev/null
@@ -1,110 +0,0 @@
-apiVersion: kubeflow.org/v1alpha2
-kind: MPIJob
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  slotsPerWorker: 1
-  cleanPodPolicy: Running
-  mpiReplicaSpecs:
-    Launcher:
-      replicas: 1
-      template:
-        spec:
-          serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-          containers:
-          - name: mpi-launcher
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-            command:
-            - mpirun
-            args:
-            - --allow-run-as-root
-            - -np
-            - "2"
-            - -x
-            - PYTHONPATH
-            - -x
-            - LD_LIBRARY_PATH
-            - -mca
-            - pml
-            - ob1
-            - -mca
-            - btl
-            - ^openib
-            - -mca
-            - btl_tcp_if_exclude
-            - lo,docker0
-            - -bind-to
-            - none
-            - -map-by
-            - slot
-            - python3
-            - /workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py # {"$openapi":"MODEL_SCRIPT"}
-            - --task_name=MRPC
-            - --do_train=true
-            - --do_eval=true
-            - --data_dir=$(DATASET_DIR)/$(GLUE_DIR)/MRPC
-            - --vocab_file=$(DATASET_DIR)/$(BERT_BASE_DIR)/vocab.txt
-            - --bert_config_file=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_config.json
-            - --init_checkpoint=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_model.ckpt
-            - --max_seq_length=128
-            - --train_batch_size=32
-            - --learning_rate=2e-5
-            - --num_train_epochs=3.0
-            - --output_dir=$(OUTPUT_DIR)
-            - --use_tpu=False
-            - --precision=fp32
-            - --do_lower_case=True
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: nfs-path
-              mountPath: /nfs # {"$openapi":"NFS_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: nfs-path
-            nfs:
-              server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-              path: /nfs # {"$openapi":"NFS_PATH"}
-    Worker:
-      replicas: 2 # {"$openapi":"WORKERS"}
-      template:
-        spec:
-          serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-          containers:
-          - name: mpi-worker
-            imagePullPolicy: IfNotPresent
-            workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-            image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-            securityContext:
-              runAsUser: 0 # {"$openapi":"USER_ID"}
-              runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-              fsGroup: 0 # {"$openapi":"FS_ID"}
-            envFrom:
-            - configMapRef:
-                name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-            volumeMounts:
-            - name: datasets
-              mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-              readOnly: true
-            - name: nfs-path
-              mountPath: /nfs # {"$openapi":"NFS_PATH"}
-          volumes:
-          - name: datasets
-            hostPath:
-              path: /datasets # {"$openapi":"DATASET_DIR"}
-          - name: nfs-path
-            nfs:
-              server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-              path: /nfs # {"$openapi":"NFS_PATH"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index b26e65897..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/multi-node/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
deleted file mode 100644
index c46cf4d7a..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,236 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.BERT_BASE_DIR:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: BERT_BASE_DIR
-          value: bert_official/MRPC/uncased_L-12_H-768_A-12
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GLUE_DIR:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: GLUE_DIR
-          value: bert_official
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: bert-large-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/language-modeling:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/bert-large-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training_single_node.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint directory
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: /checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
-    io.k8s.cli.substitutions.PYTHONPATH:
-      x-k8s-cli:
-        substitution:
-          name: PYTHONPATH
-          pattern: ${MODEL_DIR}/benchmarks:${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index c9a83fd1a..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  CHECKPOINT_DIR: /pvc/root/bert-large-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  BERT_BASE_DIR: bert_official/MRPC/uncased_L-12_H-768_A-12 # {"$openapi":"BERT_BASE_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GLUE_DIR: bert_official # {"$openapi":"GLUE_DIR"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/bert-large-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/bert-large-fp32-training/benchmarks:/workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
deleted file mode 100644
index dd98c84cd..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
+++ /dev/null
@@ -1,88 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: create-output-dir
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(OUTPUT_DIR)
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  - name: create-checkpoint-dir
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/bert-large-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=bert_large
-    - --precision=fp32
-    - --mode=training
-    - --framework=tensorflow
-    - --batch-size
-    - "32"
-    - --output-dir=$(OUTPUT_DIR)
-    - --
-    - train-option=Classifier
-    - task-name=MRPC
-    - do-train=true
-    - do-eval=true
-    - data-dir=$(DATASET_DIR)/$(GLUE_DIR)/MRPC
-    - vocab-file=$(DATASET_DIR)/$(BERT_BASE_DIR)/vocab.txt
-    - config-file=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_config.json
-    - init-checkpoint=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_model.ckpt
-    - max-seq-length=128
-    - learning-rate=2e-5
-    - num-train-epochs=30
-    - optimized_softmax=True
-    - experimental_gelu=False
-    - do-lower-case=True
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: pvc-path
-    persistentVolumeClaim:
-      claimName: workdisk # {"$openapi":"PVC_NAME"}
-  restartPolicy: OnFailure
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index b26e65897..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
deleted file mode 100644
index b4debd5ea..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,236 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.BERT_BASE_DIR:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: BERT_BASE_DIR
-          value: bert_official/MRPC/uncased_L-12_H-768_A-12
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GLUE_DIR:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: GLUE_DIR
-          value: bert_official
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: bert-large-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/language-modeling:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/bert-large-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training_single_node.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint directory
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: /checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
-    io.k8s.cli.substitutions.PYTHONPATH:
-      x-k8s-cli:
-        substitution:
-          name: PYTHONPATH
-          pattern: ${MODEL_DIR}/benchmarks:${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 41cc850bb..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,18 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  CHECKPOINT_DIR: /nfs/root/bert-large-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  BERT_BASE_DIR: bert_official/MRPC/uncased_L-12_H-768_A-12 # {"$openapi":"BERT_BASE_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GLUE_DIR: bert_official # {"$openapi":"GLUE_DIR"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/bert-large-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/bert-large-fp32-training/benchmarks:/workspace/bert-large-fp32-training/models/language_modeling/tensorflow/bert_large/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
deleted file mode 100644
index 5588ed2d2..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
+++ /dev/null
@@ -1,89 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: create-output-dir
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(OUTPUT_DIR)
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  - name: create-checkpoint-dir
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/language-modeling:tf-latest-bert-large-fp32-training # {"$openapi":"IMAGE"}
-    workingDir: /workspace/bert-large-fp32-training # {"$openapi":"MODEL_DIR"}
-    envFrom:
-    - configMapRef:
-        name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/bert-large-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=bert_large
-    - --precision=fp32
-    - --mode=training
-    - --framework=tensorflow
-    - --batch-size
-    - "32"
-    - --output-dir=$(OUTPUT_DIR)
-    - --
-    - train-option=Classifier
-    - task-name=MRPC
-    - do-train=true
-    - do-eval=true
-    - data-dir=$(DATASET_DIR)/$(GLUE_DIR)/MRPC
-    - vocab-file=$(DATASET_DIR)/$(BERT_BASE_DIR)/vocab.txt
-    - config-file=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_config.json
-    - init-checkpoint=$(DATASET_DIR)/$(BERT_BASE_DIR)/bert_model.ckpt
-    - max-seq-length=128
-    - learning-rate=2e-5
-    - num-train-epochs=30
-    - optimized_softmax=True
-    - experimental_gelu=False
-    - do-lower-case=True
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: nfs-path
-    nfs:
-      server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-      path: /nfs # {"$openapi":"NFS_PATH"}
-  restartPolicy: OnFailure
diff --git a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml b/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index b26e65897..000000000
--- a/k8s/language_modeling/tensorflow/bert_large/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: bert-large-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/datasets.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/datasets.md
deleted file mode 100644
index f5b5b22d8..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/description.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/description.md
deleted file mode 100644
index f9098baed..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/description.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/download.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/kubernetes.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/kubernetes.md
deleted file mode 100644
index bdd3e9f16..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/kubernetes.md
+++ /dev/null
@@ -1,253 +0,0 @@
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the RFCN FP32 inference package.
-```
-wget <package url>
-tar -xvf <package name>
-```
-
-The Kubernetes* package for `<model name> <precision> <mode>` includes serving and pipeline kubernetes deployments.
-Within the serving and pipeline deployments are common use cases that include storage and security
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below, 
-where serving and pipeline directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-
-```
-quickstart
-└── mlops
-    ├── pipeline
-    │       ├── user-allocated-pvc
-    │       └── user-mounted-nfs
-    └── serving
-            ├── user-allocated-pvc
-            └── user-mounted-nfs
-```
-
-
-The `pipeline` example can be used to preprocess the coco dataset to get a
-TF records file and then run an RFCN FP32 accuracy test using an
-[argo workflow](https://github.com/argoproj/argo). Deployment of argo needs to be done by devops.
-
-The `serving` example uses a pod to run inference to get performance metrics (using raw
-images from the coco dataset) or test accuracy (when you already have the TF records file on NFS).
-
-The deployments use [kustomize](https://kustomize.io/) to configure
-parameters. The parameters can be set by running kustomize commands
-prior to deploying the job to kubernetes.
-
-#### Prerequisites
-
-The <package name> package uses [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure parameters within the deployment.yaml. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### Serving Inference
-
-Inference is run by submitting a pod yaml file to the k8s api-server,
-which results in the pod creation and then the specified
-[quickstart script](#quick-start-scripts) is run in the pod's container.
-
-Make sure you are inside the serving directory:
-
-```
-cd <package dir>/quickstart/mlops/serving
-```
-
-The parameters that can be changed within the serving deployment are shown in the table below:
-
-|     NAME     |             VALUE              |         DESCRIPTION         |
-|--------------|--------------------------------|-----------------------------|
-| DATASET_DIR  | /datasets                      | input dataset directory     |
-| FS_ID        | 0                              | owner id of mounted volumes |
-| GROUP_ID     | 0                              | process group id            |
-| GROUP_NAME   | root                           | process group name          |
-| NFS_PATH     | /nfs                           | nfs path                    |
-| NFS_SERVER   | 0.0.0.0                        | nfs server                  |
-| PVC_NAME     | workdisk                       | pvc name                    |
-| PVC_PATH     | /pvc                           | pvc path                    |
-| OUTPUT_DIR   | output                         | output dir basename         |
-| USER_ID      | 0                              | process owner id            |
-| USER_NAME    | root                           | process owner name          |
-
-> Note that when running inference, the `DATASET_DIR` should point to the
-> directory of raw coco images (val2017) and when running accuracy testing,
-> the `DATASET_DIR` should point to the TF records directory.
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Serving inference output
-
-Viewing the log output of the <package name> is done by viewing the logs of the
-deployed pod. This pod is found by filtering the list of pods for the name 'inference'
-
-```
-kubectl get pods -oname|grep inference|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the inference job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep inference|cut -c5-)
-```
-
-##### Serving inference cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Pipeline
-
-The pipeline job uses an [Argo workflow](https://github.com/argoproj/argo)
-to first convert the raw coco images to the TF records format and then
-runs RFCN FP32 inference with an accuracy test using the TF records file.
-
-The [COCO validation 2017 dataset and annotations](https://cocodataset.org/#download)
-need to be downloaded to a directory on nfs. These will be used to create
-the TF records file.
-
-Make sure you are inside the pipeline directory:
-
-```
-cd <package dir>/quickstart/mlops/pipeline
-```
-
-The parameters that can be changed within the pipeline are shown in the table below:
-
-|       NAME        |             VALUE              |          DESCRIPTION           |
-|-------------------|--------------------------------|--------------------------------|
-| DATASET_DIR       | /datasets                      | input dataset directory        |
-| FS_ID             | 0                              | owner id of mounted volumes    |
-| GROUP_ID          | 0                              | process group id               |
-| GROUP_NAME        | root                           | process group name             |
-| NFS_PATH          | /nfs                           | nfs path                       |
-| NFS_SERVER        | 0.0.0.0                        | nfs server                     |
-| PVC_NAME          | workdisk                       | pvc name                       |
-| PVC_PATH          | /pvc                           | pvc path                       |
-| OUTPUT_DIR        | output                         | output dir basename            |
-| USER_ID           | 0                              | process owner id               |
-| USER_NAME         | root                           | process owner name             |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity[^3].
-
-[^3]: In order for the argo workflow to run as a non root user it must set the WorkflowExecutor to be k8sapi, otherwise the workflow will fail with "Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock". See argo issues [2239](https://github.com/argoproj/argo/issues/2239),[4186](https://github.com/argoproj/argo/issues/4186). Setting argo's WorkflowExecutor to k8sapi is described [here](https://argoproj.github.io/argo/workflow-executors/). This must be performed by devops.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-Once the job has been submitted, the status and logs can be viewed using
-the Argo user inferface or from the command line using kubectl or argo.
-The commands below describe how to use kubectl to see the workflow, pods,
-and log files:
-```
-kubectl get wf
-kubectl get pods
-kubectl logs <pod name> main
-```
-
-##### Cleanup
-
-Remove the workflow and related resources using the following command:
-
-```
-kubectl delete -f <use case>.yaml
-```
-
-### Advanced Options
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/license.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/quickstart.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/quickstart.md
deleted file mode 100644
index d61cb3270..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_inference.sh`](mlops/serving/user-mounted-nfs/pod.yaml#L16) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`fp32_accuracy.sh`](mlops/pipeline/user-mounted-nfs/serving_accuracy.yaml#L49) | Processes the TF records to run inference and check accuracy on the results. |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/title.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/troubleshooting.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/troubleshooting.md
deleted file mode 100644
index fd80903fb..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.docs/troubleshooting.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/deployments-yml-exist.sh b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/deployments-yml-exist.sh
deleted file mode 100644
index f6aabf8b3..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/deployments-yml-exist.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-load "lib/test_helper"
-
-@test "validate package for rfcn-fp32-inference-k8s in framework k8s creates model package file" {
-  run model-builder package -f k8s rfcn-fp32-inference-k8s
-  (( $status == 0 ))
-  [[ -f output/rfcn-fp32-inference-k8s.tar.gz ]]
-}
-
-@test "validate generate-deployment for rfcn-fp32-inference-k8s in framework k8s creates deployment files under deployments/rfcn-fp32-inference-k8s/" {
-  run model-builder generate-deployment -f k8s rfcn-fp32-inference-k8s
-  (( $status == 0 ))
-  [[ -f deployments/rfcn-fp32-inference-k8s/pipeline/user-allocated-pvc.yaml ]]
-  [[ -f deployments/rfcn-fp32-inference-k8s/pipeline/user-mounted-nfs.yaml ]]
-  [[ -f deployments/rfcn-fp32-inference-k8s/serving/user-allocated-pvc.yaml ]]
-  [[ -f deployments/rfcn-fp32-inference-k8s/serving/user-mounted-nfs.yaml ]]
-  (( $(last_modified deployments/rfcn-fp32-inference-k8s/pipeline/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/rfcn-fp32-inference-k8s/pipeline/user-mounted-nfs.yaml) <= 50 ))
-  (( $(last_modified deployments/rfcn-fp32-inference-k8s/serving/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/rfcn-fp32-inference-k8s/serving/user-mounted-nfs.yaml) <= 50 ))
-}
\ No newline at end of file
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/namespace-exists.sh b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/namespace-exists.sh
deleted file mode 100644
index 4a76a6981..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/namespace-exists.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "verify namespace $USER_NAME exists" {
-  run verify "there is 1 namespace named '"$USER_NAME"'"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/serving_deployment.sh b/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/serving_deployment.sh
deleted file mode 100644
index 8bcf8d25d..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/.tests/serving_deployment.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "kubectl apply of rfcn-fp32-inference-k8s serving" {
-  run $DETIK_CLIENT_NAME apply -f deployments/rfcn-fp32-inference-k8s/serving/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "deployment of rfcn-fp32-inference-k8s serving creates 1 pod and it is running" {
-  try "at most 4 times every 30s to get pods named 'rfcn-fp32-inference' and verify that 'status' is 'running'"
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "verify job completion, check end of output includes text 'run_final'" {
-  run launcher_logs
-  echo "output = ${output}"
-  (( $status == 0 ))
-  [[ ${lines[4]} =~ run_final ]]
-}
-
-@test "kubectl delete of rfcn-fp32-inference-k8s serving" {
-  run $DETIK_CLIENT_NAME delete -f deployments/rfcn-fp32-inference-k8s/serving/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/README.md b/k8s/object_detection/tensorflow/rfcn/inference/fp32/README.md
deleted file mode 100644
index a4e3fe2f0..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/README.md
+++ /dev/null
@@ -1,312 +0,0 @@
-<!--- 0. Title -->
-# RFCN FP32 inference
-
-<!-- 10. Description -->
-
-This document has instructions for running RFCN FP32 inference using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-<!--- 20. Download link -->
-## Download link
-
-[rfcn-fp32-inference-k8s.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/rfcn-fp32-inference-k8s.tar.gz)
-
-<!--- 30. Datasets -->
-## Dataset
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_inference.sh`](mlops/serving/user-mounted-nfs/pod.yaml#L16) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`fp32_accuracy.sh`](mlops/pipeline/user-mounted-nfs/serving_accuracy.yaml#L49) | Processes the TF records to run inference and check accuracy on the results. |
-
-These quickstart scripts can be run in the following environment:
-* [Kubernetes](#kubernetes)
-
-
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the RFCN FP32 inference package.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/rfcn-fp32-inference-k8s.tar.gz
-tar -xvf rfcn-fp32-inference-k8s.tar.gz
-```
-
-The Kubernetes* package for `RFCN FP32 inference` includes serving and pipeline kubernetes deployments.
-Within the serving and pipeline deployments are common use cases that include storage and security
-variations that are common across different kubernetes installations. The directory tree within the model package is shown below, 
-where serving and pipeline directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-
-```
-quickstart
-└── mlops
-    ├── pipeline
-    │       ├── user-allocated-pvc
-    │       └── user-mounted-nfs
-    └── serving
-            ├── user-allocated-pvc
-            └── user-mounted-nfs
-```
-
-
-The `pipeline` example can be used to preprocess the coco dataset to get a
-TF records file and then run an RFCN FP32 accuracy test using an
-[argo workflow](https://github.com/argoproj/argo). Deployment of argo needs to be done by devops.
-
-The `serving` example uses a pod to run inference to get performance metrics (using raw
-images from the coco dataset) or test accuracy (when you already have the TF records file on NFS).
-
-The deployments use [kustomize](https://kustomize.io/) to configure
-parameters. The parameters can be set by running kustomize commands
-prior to deploying the job to kubernetes.
-
-#### Prerequisites
-
-The rfcn-fp32-inference-k8s.tar.gz package uses [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure parameters within the deployment.yaml. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### Serving Inference
-
-Inference is run by submitting a pod yaml file to the k8s api-server,
-which results in the pod creation and then the specified
-[quickstart script](#quick-start-scripts) is run in the pod's container.
-
-Make sure you are inside the serving directory:
-
-```
-cd rfcn-fp32-inference-k8s/quickstart/mlops/serving
-```
-
-The parameters that can be changed within the serving deployment are shown in the table below:
-
-|     NAME     |             VALUE              |         DESCRIPTION         |
-|--------------|--------------------------------|-----------------------------|
-| DATASET_DIR  | /datasets                      | input dataset directory     |
-| FS_ID        | 0                              | owner id of mounted volumes |
-| GROUP_ID     | 0                              | process group id            |
-| GROUP_NAME   | root                           | process group name          |
-| NFS_PATH     | /nfs                           | nfs path                    |
-| NFS_SERVER   | 0.0.0.0                        | nfs server                  |
-| PVC_NAME     | workdisk                       | pvc name                    |
-| PVC_PATH     | /pvc                           | pvc path                    |
-| OUTPUT_DIR   | output                         | output dir basename         |
-| USER_ID      | 0                              | process owner id            |
-| USER_NAME    | root                           | process owner name          |
-
-> Note that when running inference, the `DATASET_DIR` should point to the
-> directory of raw coco images (val2017) and when running accuracy testing,
-> the `DATASET_DIR` should point to the TF records directory.
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Serving inference output
-
-Viewing the log output of the rfcn-fp32-inference-k8s.tar.gz is done by viewing the logs of the
-deployed pod. This pod is found by filtering the list of pods for the name 'inference'
-
-```
-kubectl get pods -oname|grep inference|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the inference job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep inference|cut -c5-)
-```
-
-##### Serving inference cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Pipeline
-
-The pipeline job uses an [Argo workflow](https://github.com/argoproj/argo)
-to first convert the raw coco images to the TF records format and then
-runs RFCN FP32 inference with an accuracy test using the TF records file.
-
-The [COCO validation 2017 dataset and annotations](https://cocodataset.org/#download)
-need to be downloaded to a directory on nfs. These will be used to create
-the TF records file.
-
-Make sure you are inside the pipeline directory:
-
-```
-cd rfcn-fp32-inference-k8s/quickstart/mlops/pipeline
-```
-
-The parameters that can be changed within the pipeline are shown in the table below:
-
-|       NAME        |             VALUE              |          DESCRIPTION           |
-|-------------------|--------------------------------|--------------------------------|
-| DATASET_DIR       | /datasets                      | input dataset directory        |
-| FS_ID             | 0                              | owner id of mounted volumes    |
-| GROUP_ID          | 0                              | process group id               |
-| GROUP_NAME        | root                           | process group name             |
-| NFS_PATH          | /nfs                           | nfs path                       |
-| NFS_SERVER        | 0.0.0.0                        | nfs server                     |
-| PVC_NAME          | workdisk                       | pvc name                       |
-| PVC_PATH          | /pvc                           | pvc path                       |
-| OUTPUT_DIR        | output                         | output dir basename            |
-| USER_ID           | 0                              | process owner id               |
-| USER_NAME         | root                           | process owner name             |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity[^3].
-
-[^3]: In order for the argo workflow to run as a non root user it must set the WorkflowExecutor to be k8sapi, otherwise the workflow will fail with "Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock". See argo issues [2239](https://github.com/argoproj/argo/issues/2239),[4186](https://github.com/argoproj/argo/issues/4186). Setting argo's WorkflowExecutor to k8sapi is described [here](https://argoproj.github.io/argo/workflow-executors/). This must be performed by devops.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-Once the job has been submitted, the status and logs can be viewed using
-the Argo user inferface or from the command line using kubectl or argo.
-The commands below describe how to use kubectl to see the workflow, pods,
-and log files:
-```
-kubectl get wf
-kubectl get pods
-kubectl logs <pod name> main
-```
-
-##### Cleanup
-
-Remove the workflow and related resources using the following command:
-
-```
-kubectl delete -f object_detection.yaml
-```
-
-### Advanced Options
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
-
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/Krmfile b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/Krmfile
deleted file mode 100644
index 52f21fa55..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,280 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.substitutions.ANNOTATIONS_DIR:
-      x-k8s-cli:
-        substitution:
-          name: ANNOTATIONS_DIR
-          pattern: ${DATASET_DIR}/annotations
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.PREPROCESS_IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_IMAGE_VERSION
-          value: tf-1.15.2
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/rfcn-fp32-inference
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: rfcn-fp32-inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_accuracy.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.PREPROCESS_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: PREPROCESS_COMMAND
-          pattern: ${PREPROCESS_DIR}/scripts/${PREPROCESS_SCRIPT}
-          values:
-          - marker: ${PREPROCESS_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_DIR'
-          - marker: ${PREPROCESS_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_SCRIPT'
-    io.k8s.cli.setters.PREPROCESS_DIR:
-      description: container preprocess directory
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_DIR
-          value: /workspace/preprocess-coco-val
-          setBy: model-builder
-    io.k8s.cli.substitutions.PREPROCESS_IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: PREPROCESS_IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${PREPROCESS_IMAGE_VERSION}-${PREPROCESS_NAME}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${PREPROCESS_IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_IMAGE_VERSION'
-          - marker: ${PREPROCESS_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.PREPROCESS_NAME:
-      description: preprocess image part
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_NAME
-          value: preprocess-coco-val
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.PREPROCESS_SCRIPT:
-      description: preprocess script
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_SCRIPT
-          value: preprocess_coco_val.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.VAL_IMAGE_DIR:
-      x-k8s-cli:
-        substitution:
-          name: VAL_IMAGE_DIR
-          pattern: ${DATASET_DIR}/val2017
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index d1d16b300..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,14 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-data:
-  ANNOTATIONS_DIR: /datasets/annotations # {"$openapi":"ANNOTATIONS_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  OUTPUT_DIR: /pvc/root/rfcn-fp32-inference/output # {"$openapi":"OUTPUT_PATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
-  VAL_IMAGE_DIR: /datasets/val2017 # {"$openapi":"VAL_IMAGE_DIR"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index 213b180d6..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- role.yaml
-- role-binding.yaml
-- serving_accuracy.yaml
-namespace: default
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml
deleted file mode 100644
index 460d60d00..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-subjects:
-- kind: ServiceAccount
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role.yaml
deleted file mode 100644
index 15db50fb2..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/role.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  - configmaps
-  - services
-  verbs:
-  - get
-  - list
-  - create
-  - update
-  - delete
-  - watch
-  - patch
-- apiGroups:
-  - ""
-  resources:
-  - pods/log
-  verbs:
-  - get
-  - watch
-- apiGroups:
-  - extensions
-  - apps
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 6149957d5..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/serving_accuracy.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/serving_accuracy.yaml
deleted file mode 100644
index c81f9ac48..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-allocated-pvc/serving_accuracy.yaml
+++ /dev/null
@@ -1,65 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Workflow
-metadata:
-  name: rfcn-fp32-inference-wf # {"$openapi":"WORKFLOW_NAME"}
-spec:
-  entrypoint: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  templates:
-  - name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    steps:
-    - - name: preprocess-coco
-        template: preprocess-coco
-    - - name: rfcn-fp32-accuracy
-        template: rfcn-fp32-accuracy
-  - name: preprocess-coco
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/object-detection:tf-1.15.2-preprocess-coco-val # {"$openapi":"PREPROCESS_IMAGE"}
-      command: # {"$openapi":"PREPROCESS_COMMAND"}
-      - /workspace/preprocess-coco-val/scripts/preprocess_coco_val.sh # {"$openapi":"PREPROCESS_COMMAND"}
-      envFrom:
-      - configMapRef:
-          name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: pvc-path
-        mountPath: /pvc # {"$openapi":"PVC_PATH"}
-      - name: coco-raw-data
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-    volumes:
-    - name: coco-raw-data
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: pvc-path
-      persistentVolumeClaim:
-        claimName: workdisk # {"$openapi":"PVC_NAME"}
-  - name: rfcn-fp32-accuracy
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-      imagePullPolicy: Always
-      command: # {"$openapi":"MODEL_COMMAND"}
-      - /workspace/rfcn-fp32-inference/quickstart/fp32_accuracy.sh # {"$openapi":"MODEL_COMMAND"}
-      envFrom:
-      - configMapRef:
-          name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: coco-raw-data
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-      - name: pvc-path
-        mountPath: /pvc # {"$openapi":"PVC_PATH"}
-    volumes:
-    - name: coco-raw-data
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: pvc-path
-      persistentVolumeClaim:
-        claimName: workdisk # {"$openapi":"PVC_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/Krmfile b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/Krmfile
deleted file mode 100644
index 5a10130cb..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,262 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.substitutions.ANNOTATIONS_DIR:
-      x-k8s-cli:
-        substitution:
-          name: ANNOTATIONS_DIR
-          pattern: ${DATASET_DIR}/annotations
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/rfcn-fp32-inference
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: rfcn-fp32-inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_accuracy.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.substitutions.PREPROCESS_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: PREPROCESS_COMMAND
-          pattern: ${PREPROCESS_DIR}/scripts/${PREPROCESS_SCRIPT}
-          values:
-          - marker: ${PREPROCESS_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_DIR'
-          - marker: ${PREPROCESS_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_SCRIPT'
-    io.k8s.cli.setters.PREPROCESS_DIR:
-      description: container preprocess directory
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_DIR
-          value: /workspace/preprocess-coco-val
-          setBy: model-builder
-    io.k8s.cli.setters.PREPROCESS_IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_IMAGE_VERSION
-          value: tf-1.15.2
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.PREPROCESS_IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: PREPROCESS_IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${PREPROCESS_IMAGE_VERSION}-${PREPROCESS_NAME}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${PREPROCESS_IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_IMAGE_VERSION'
-          - marker: ${PREPROCESS_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.PREPROCESS_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.PREPROCESS_NAME:
-      description: preprocess image part
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_NAME
-          value: preprocess-coco-val
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.PREPROCESS_SCRIPT:
-      description: preprocess script
-      x-k8s-cli:
-        setter:
-          name: PREPROCESS_SCRIPT
-          value: preprocess_coco_val.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.VAL_IMAGE_DIR:
-      x-k8s-cli:
-        substitution:
-          name: VAL_IMAGE_DIR
-          pattern: ${DATASET_DIR}/val2017
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index d1e11fb7b..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,14 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-data:
-  ANNOTATIONS_DIR: /datasets/annotations # {"$openapi":"ANNOTATIONS_DIR"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  OUTPUT_DIR: /nfs/root/rfcn-fp32-inference/output # {"$openapi":"OUTPUT_PATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
-  VAL_IMAGE_DIR: /datasets/val2017 # {"$openapi":"VAL_IMAGE_DIR"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index 213b180d6..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- role.yaml
-- role-binding.yaml
-- serving_accuracy.yaml
-namespace: default
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml
deleted file mode 100644
index 460d60d00..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-subjects:
-- kind: ServiceAccount
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role.yaml
deleted file mode 100644
index 15db50fb2..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/role.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  - configmaps
-  - services
-  verbs:
-  - get
-  - list
-  - create
-  - update
-  - delete
-  - watch
-  - patch
-- apiGroups:
-  - ""
-  resources:
-  - pods/log
-  verbs:
-  - get
-  - watch
-- apiGroups:
-  - extensions
-  - apps
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 6149957d5..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/serving_accuracy.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/serving_accuracy.yaml
deleted file mode 100644
index d01d239b0..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/pipeline/user-mounted-nfs/serving_accuracy.yaml
+++ /dev/null
@@ -1,67 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Workflow
-metadata:
-  name: rfcn-fp32-inference-wf # {"$openapi":"WORKFLOW_NAME"}
-spec:
-  entrypoint: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  serviceAccountName: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  templates:
-  - name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    steps:
-    - - name: preprocess-coco
-        template: preprocess-coco
-    - - name: rfcn-fp32-accuracy
-        template: rfcn-fp32-accuracy
-  - name: preprocess-coco
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/object-detection:tf-1.15.2-preprocess-coco-val # {"$openapi":"PREPROCESS_IMAGE"}
-      command: # {"$openapi":"PREPROCESS_COMMAND"}
-      - /workspace/preprocess-coco-val/scripts/preprocess_coco_val.sh # {"$openapi":"PREPROCESS_COMMAND"}
-      envFrom:
-      - configMapRef:
-          name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: nfs-path
-        mountPath: /nfs # {"$openapi":"NFS_PATH"}
-      - name: coco-raw-data
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-    volumes:
-    - name: coco-raw-data
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: nfs-path
-      nfs:
-        server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-        path: /nfs # {"$openapi":"NFS_PATH"}
-  - name: rfcn-fp32-accuracy
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-      imagePullPolicy: Always
-      command: # {"$openapi":"MODEL_COMMAND"}
-      - /workspace/rfcn-fp32-inference/quickstart/fp32_accuracy.sh # {"$openapi":"MODEL_COMMAND"}
-      envFrom:
-      - configMapRef:
-          name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: nfs-path
-        mountPath: /nfs # {"$openapi":"NFS_PATH"}
-      - name: coco-raw-data
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-    volumes:
-    - name: coco-raw-data
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: nfs-path
-      nfs:
-        server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-        path: /nfs # {"$openapi":"NFS_PATH"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/Krmfile b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/Krmfile
deleted file mode 100644
index e50df8e1d..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,262 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/rfcn-fp32-inference
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: rfcn-fp32-inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PRETRAINED_MODEL_DIR:
-      description: pretrained model location
-      x-k8s-cli:
-        setter:
-          name: PRETRAINED_MODEL_DIR
-          value: pretrained_model
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.PRETRAINED_MODEL_PATH:
-      x-k8s-cli:
-        substitution:
-          name: PRETRAINED_MODEL_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${PRETRAINED_MODEL_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${PRETRAINED_MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PRETRAINED_MODEL_DIR'
-    io.k8s.cli.setters.FROZEN_GRAPH:
-      description: frozen graph
-      x-k8s-cli:
-        setter:
-          name: FROZEN_GRAPH
-          value: frozen_inference_graph.pb
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.FROZEN_GRAPH_PATH:
-      x-k8s-cli:
-        substitution:
-          name: FROZEN_GRAPH_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${PRETRAINED_MODEL_DIR}/rfcn_resnet101_coco_2018_01_28/${FROZEN_GRAPH}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${PRETRAINED_MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PRETRAINED_MODEL_DIR'
-          - marker: ${FROZEN_GRAPH}
-            ref: '#/definitions/io.k8s.cli.setters.FROZEN_GRAPH'
-    io.k8s.cli.substitutions.VAL_IMAGE_PATH:
-      x-k8s-cli:
-        substitution:
-          name: VAL_IMAGE_PATH
-          pattern: ${DATASET_DIR}/val2017
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/config-map.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index 13d8adabd..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FROZEN_GRAPH: /pvc/root/rfcn-fp32-inference/pretrained_model/rfcn_resnet101_coco_2018_01_28/frozen_inference_graph.pb # {"$openapi":"FROZEN_GRAPH_PATH"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  OUTPUT_DIR: /pvc/root/rfcn-fp32-inference/output # {"$openapi":"OUTPUT_PATH"}
-  PRETRAINED_MODEL_DIR: /pvc/root/rfcn-fp32-inference/pretrained_model # {"$openapi":"PRETRAINED_MODEL_PATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
-  VAL_IMAGE_DIR: /datasets/val2017 # {"$openapi":"VAL_IMAGE_PATH"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/kustomization.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/pod.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/pod.yaml
deleted file mode 100644
index a97957d44..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/pod.yaml
+++ /dev/null
@@ -1,74 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: extract-pretrained-model
-    image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/rfcn-fp32-inference # {"$openapi":"MODEL_DIR"}
-    command:
-    - bash
-    - -c
-    args:
-    - mkdir -p $(PRETRAINED_MODEL_DIR);
-      find $(PRETRAINED_MODEL_DIR) -type d -exec chmod 777 {} \;;
-      tar -C $(PRETRAINED_MODEL_DIR) -xvf pretrained_model/rfcn_fp32_model.tar.gz;
-    envFrom:
-    - configMapRef:
-        name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  containers:
-  - name: serving
-    image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-    workingDir: /workspace/rfcn-fp32-inference # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/rfcn-fp32-inference/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name
-    - rfcn
-    - --mode
-    - inference
-    - --precision
-    - fp32
-    - --framework
-    - tensorflow
-    - --model-source-dir
-    - /tensorflow/models
-    - --data-location
-    - $(VAL_IMAGE_DIR)
-    - --in-graph
-    - $(FROZEN_GRAPH)
-    - --output-dir
-    - $(OUTPUT_DIR)
-    - --batch-size
-    - "1"
-    - --benchmark-only
-    - --
-    - number_of_steps=500
-    envFrom:
-    - configMapRef:
-        name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: pvc-path
-    persistentVolumeClaim:
-      claimName: workdisk # {"$openapi":"PVC_NAME"}
-  restartPolicy: OnFailure
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/service-account.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 6149957d5..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/Krmfile b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/Krmfile
deleted file mode 100644
index ec5002039..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,244 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/object-detection:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/rfcn-fp32-inference
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: rfcn-fp32-inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_inference
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.PRETRAINED_MODEL_DIR:
-      description: pretrained model location
-      x-k8s-cli:
-        setter:
-          name: PRETRAINED_MODEL_DIR
-          value: pretrained_model
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.PRETRAINED_MODEL_PATH:
-      x-k8s-cli:
-        substitution:
-          name: PRETRAINED_MODEL_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${PRETRAINED_MODEL_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${PRETRAINED_MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PRETRAINED_MODEL_DIR'
-    io.k8s.cli.setters.FROZEN_GRAPH:
-      description: frozen graph
-      x-k8s-cli:
-        setter:
-          name: FROZEN_GRAPH
-          value: frozen_inference_graph.pb
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.FROZEN_GRAPH_PATH:
-      x-k8s-cli:
-        substitution:
-          name: FROZEN_GRAPH_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${PRETRAINED_MODEL_DIR}/rfcn_resnet101_coco_2018_01_28/${FROZEN_GRAPH}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${PRETRAINED_MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.PRETRAINED_MODEL_DIR'
-          - marker: ${FROZEN_GRAPH}
-            ref: '#/definitions/io.k8s.cli.setters.FROZEN_GRAPH'
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.VAL_IMAGE_PATH:
-      x-k8s-cli:
-        substitution:
-          name: VAL_IMAGE_PATH
-          pattern: ${DATASET_DIR}/val2017
-          values:
-          - marker: ${DATASET_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.DATASET_DIR'
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/config-map.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 84d407b2a..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FROZEN_GRAPH: /nfs/root/rfcn-fp32-inference/pretrained_model/rfcn_resnet101_coco_2018_01_28/frozen_inference_graph.pb # {"$openapi":"FROZEN_GRAPH_PATH"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  OUTPUT_DIR: /nfs/root/rfcn-fp32-inference/output # {"$openapi":"OUTPUT_PATH"}
-  PRETRAINED_MODEL_DIR: /nfs/root/rfcn-fp32-inference/pretrained_model # {"$openapi":"PRETRAINED_MODEL_PATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
-  VAL_IMAGE_DIR: /datasets/val2017 # {"$openapi":"VAL_IMAGE_PATH"}
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/kustomization.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/pod.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/pod.yaml
deleted file mode 100644
index 2ed8b620a..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/pod.yaml
+++ /dev/null
@@ -1,75 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: extract-pretrained-model
-    image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/rfcn-fp32-inference # {"$openapi":"MODEL_DIR"}
-    command:
-    - bash
-    - -c
-    args:
-    - mkdir -p $(PRETRAINED_MODEL_DIR);
-      find $(PRETRAINED_MODEL_DIR) -type d -exec chmod 777 {} \;;
-      tar -C $(PRETRAINED_MODEL_DIR) -xvf pretrained_model/rfcn_fp32_model.tar.gz;
-    envFrom:
-    - configMapRef:
-        name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  containers:
-  - name: serving
-    image: docker.io/intel/object-detection:tf-latest-rfcn-fp32-inference # {"$openapi":"IMAGE"}
-    workingDir: /workspace/rfcn-fp32-inference # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/rfcn-fp32-inference/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name
-    - rfcn
-    - --mode
-    - inference
-    - --precision
-    - fp32
-    - --framework
-    - tensorflow
-    - --model-source-dir
-    - /tensorflow/models
-    - --data-location
-    - $(VAL_IMAGE_DIR)
-    - --in-graph
-    - $(FROZEN_GRAPH)
-    - --output-dir
-    - $(OUTPUT_DIR)
-    - --batch-size
-    - "1"
-    - --benchmark-only
-    - --
-    - number_of_steps=500
-    envFrom:
-    - configMapRef:
-        name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: nfs-path
-    nfs:
-      server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-      path: /nfs # {"$openapi":"NFS_PATH"}
-  restartPolicy: OnFailure
diff --git a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/service-account.yaml b/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 6149957d5..000000000
--- a/k8s/object_detection/tensorflow/rfcn/inference/fp32/mlops/serving/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: rfcn-fp32-inference # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/datasets.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/datasets.md
deleted file mode 100644
index 26e5d7dcd..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,17 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-The large [Kaggle Display Advertising Challenge Dataset](https://www.kaggle.com/c/criteo-display-ad-challenge/data)
-will be used for training Wide and Deep. The
-[data](https://www.kaggle.com/c/criteo-display-ad-challenge/data) is from
-[Criteo](https://www.criteo.com) and has a field indicating if an ad was
-clicked (1) or not (0), along with integer and categorical features.
-
-Download large Kaggle Display Advertising Challenge Dataset from
-[Criteo Labs](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/).
-* Download the large version of train dataset from: https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/train.csv
-* Download the large version of evaluation dataset from: https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/eval.csv
-
-The directory where you've downloaded the `train.csv` and `eval.csv`
-files should be used as the `DATASET_DIR` when running [quickstart scripts](#quick-start-scripts).
-
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/description.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/description.md
deleted file mode 100644
index 4f58aa953..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/description.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running [Wide and Deep](https://arxiv.org/pdf/1606.07792.pdf) <precision> <mode> using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/download.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/kubernetes.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/kubernetes.md
deleted file mode 100644
index 6984ece85..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/kubernetes.md
+++ /dev/null
@@ -1,298 +0,0 @@
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the model <mode> package to get the yaml and config
-files for running <mode> on a single node using Kubernetes.
-```
-wget <package url>
-tar -xvf <package name>
-```
-
-### Execution
-
-The Kubernetes* package for `<model name> <precision> <mode>` includes single-node and pipeline kubernetes deployments.
-Within the single and pipeline deployments are common use cases that include storage and security variations that are common 
-across different kubernetes installations. The directory tree within the kubernetes package is shown below, where 
-single-node and pipeline directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-Common use cases are found under the single and pipeline directories:
-
-```
-quickstart
-└── mlops
-    ├── pipeline
-    │       ├── user-allocated-pvc
-    │       └── user-mounted-nfs
-    └── single-node
-            ├── user-allocated-pvc
-            └── user-mounted-nfs
-```
-
-#### Prerequisites
-
-Both single-node and pipeline deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### Single-node Training
-
-Single node training is similar to the docker use case but is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running
-the [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L50) script within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the <package name> Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Model Training and TF Serving Pipeline
-
-This pipeline runs the following steps using an Argo workflow:
-1. Train the model on a single node using the `fp32_training_check_accuracy.sh`
-   script. This script runs model training for a specified number of steps,
-   exports the saved model, and compares the accuacy against the value
-   specified in the `TARGET_ACCURACY` environment variable. If the model's
-   accuracy does not meet the target accuracy, this step will be retried
-   and continues training based on previous checkpoints in the specified
-   `CHECKPOINT_DIR`. If the `TARGET_ACCURACY` environment variable has
-   not been defined, then no accuracy check is done and it will continue
-   on to the next step, regardless of the model's accuracy.
-1. Deploy TensorFlow Serving containers with the saved model
-1. Create a service that exposes the TensorFlow Serving containers as a
-   NodePort
-
-The TensorFlow Serving steps in this pipeline follows the
-[TensorFlow Serving with Kubernetes instructions](https://www.tensorflow.org/tfx/serving/serving_kubernetes)
-with the exception that it does not use a Google Cloud Kubernetes
-cluster. Since the Kubernetes cluster being used does not have a load
-balancer, the configuration is setup for NodePort, which will allow
-external requests.
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME        |                 VALUE    |        DESCRIPTION          |
-|-----------------|--------------------------|-----------------------------|
-| DATASET_DIR     | /datasets                | input dataset directory     |
-| FS_ID           | 0                        | owner id of mounted volumes |
-| GROUP_ID        | 0                        | process group id            |
-| GROUP_NAME      | root                     | process group name          |
-| NFS_PATH        | /nfs                     | nfs path                    |
-| NFS_SERVER      | 0.0.0.0                  | nfs server                  |
-| PVC_NAME        | workdisk                 | model-builder | pvc name    |
-| PVC_PATH        | /pvc                     | model-builder | pvc path    |
-| REPLICAS        | 3                        | replica number              |
-| RETRY_LIMIT     | 10                       | replica number              |
-| TARGET_ACCURACY | 0.74                     | target accuracy             |
-| TF_SERVING_PORT | 8501                     | tf serving port             |
-| USER_ID    | 0                             | process owner id            |
-| USER_NAME  | root                          | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the argo workflow[^3] is deployed with the user's identity.
-
-[^3]: In order for the argo workflow to run as a non root user it must set the WorkflowExecutor to be k8sapi, otherwise the workflow will fail with "Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock". See argo issue [2239](https://github.com/argoproj/argo/issues/2239). Setting argo's WorkflowExecutor to k8sapi is described [here](https://argoproj.github.io/argo/workflow-executors/). This must be performed by devops.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-Once the job has been submitted, the status and logs can be viewed using
-the Argo user inferface or from the command line using kubectl or argo.
-The commands below describe how to use kubectl to see the workflow, pods,
-and log files:
-
-```
-kubectl get wf
-kubectl get pods
-kubectl logs <pod name> main
-```
-
-##### TensorFlow Serving Client
-
-Once all the steps in the workflow have completed, the TensorFlow
-Serving GRPC client can be used to run inference on the served model.
-
-Prior to running the client script, install the following dependency in
-your environment:
-* tensorflow-serving-api
-
-The client script reads a csv file (in this example we are using the
-[eval.csv file](#dataset)), formats the data in for input parameter, and
-then calls the served model. Accuracy and benchmarking metrics are
-printed out.
-
-Run the [run_tf_serving_client.py](run_tf_serving_client.py) script with
-the `--help` flag to see the argument options:
-```
-$ python run_wide_deep_client.py --help
-usage: <package dir>/quickstart/run_tf_serving_client.py [-h]
-       [-s SERVER] -d DATA_FILE [-b BATCH_SIZE] [-n NUM_ITERATION] [-w WARM_UP_ITERATION]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -s SERVER, --server SERVER
-                        Server URL and port (default=localhost:8500).
-  -d DATA_FILE, --data_file DATA_FILE
-                        Path to csv data file
-  -b BATCH_SIZE, --batch_size BATCH_SIZE
-                        Batch size to use (default=1).
-  -n NUM_ITERATION, --num_iteration NUM_ITERATION
-                        Number of times to repeat (default=40).
-  -w WARM_UP_ITERATION, --warm_up_iteration WARM_UP_ITERATION
-                        Number of initial iterations to ignore in benchmarking (default=10).
-```
-
-1. Find the `INTERNAL-IP` one of the nodes in your cluster using
-   `kubectl get nodes -o wide`. This IP should be used as the server URL
-   in the `--server` arg.
-
-1. Get the `NodePort` using `kubectl describe service`. This `NodePort`
-   should be used as the port in the `--server` arg.
-
-1. Run the client script with your preferred parameters. For example:
-   ```
-   python <package dir>/quickstart/run_tf_serving_client.py -s <Internal IP>:<Node Port> -d <path to eval.csv> --b <batch size>
-   ```
-   The script will call the served model using data from the csv file
-   and output performance and accuracy metrics.
-
-##### Clean up the pipeline
-
-To clean up the model training/serving pipeline, delete the service,
-deployment, and other resources using the following commands:
-
-```
-kubectl delete -f <use-case>.yaml
-```
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/license.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/quickstart.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/quickstart.md
deleted file mode 100644
index 0dd4a75d7..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_training_check_accuracy.sh`](mlops/pipeline/user-mounted-nfs/train_and_serve.yaml#L50) | Trains the model for a specified number of steps (default is 500) and then compare the accuracy against the specified target accuracy. If the accuracy is not met, then script exits with error code 1. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L50) | Trains the model for 10 epochs if -- steps is not specified. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-
-These quickstart scripts are run in the following environment:
-* [Kubernetes](#kubernetes)
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/title.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/troubleshooting.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/troubleshooting.md
deleted file mode 100644
index fd80903fb..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.docs/troubleshooting.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/deployments-yml-exist.sh b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/deployments-yml-exist.sh
deleted file mode 100644
index af8c62e2a..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/deployments-yml-exist.sh
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-load "lib/test_helper"
-
-@test "validate package for wide-deep-large-ds-fp32-training-k8s in framework k8s creates model package file" {
-  run model-builder package -f k8s wide-deep-large-ds-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f output/wide-deep-large-ds-fp32-training-k8s.tar.gz ]]
-}
-
-@test "validate generate-deployment for wide-deep-large-ds-fp32-training-k8s in framework k8s creates deployment files under deployments/wide-deep-large-ds-fp32-training-k8s/" {
-  run model-builder generate-deployment -f k8s wide-deep-large-ds-fp32-training-k8s
-  (( $status == 0 ))
-  [[ -f deployments/wide-deep-large-ds-fp32-training-k8s/pipeline/user-allocated-pvc.yaml ]]
-  [[ -f deployments/wide-deep-large-ds-fp32-training-k8s/pipeline/user-mounted-nfs.yaml ]]
-  [[ -f deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-allocated-pvc.yaml ]]
-  [[ -f deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-mounted-nfs.yaml ]]
-  (( $(last_modified deployments/wide-deep-large-ds-fp32-training-k8s/pipeline/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/wide-deep-large-ds-fp32-training-k8s/pipeline/user-mounted-nfs.yaml) <= 50 ))
-  (( $(last_modified deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-allocated-pvc.yaml) <= 50 ))
-  (( $(last_modified deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-mounted-nfs.yaml) <= 50 ))
-}
\ No newline at end of file
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/namespace-exists.sh b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/namespace-exists.sh
deleted file mode 100644
index 4a76a6981..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/namespace-exists.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "verify namespace $USER_NAME exists" {
-  run verify "there is 1 namespace named '"$USER_NAME"'"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/single_node_deployment.sh b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/single_node_deployment.sh
deleted file mode 100644
index c3b73043a..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/.tests/single_node_deployment.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-@test "kubectl apply of single-node wide-deep-large-ds-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME apply -f deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "single-node deployment of wide-deep-large-ds-fp32-training-k8s creates 1 pod and it is running" {
-  try "at most 4 times every 30s to get pods named 'wide-deep-large-ds-fp32-training' and verify that 'status' is 'running'"
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
-
-@test "verify job completion, check end of output includes text 'run_final'" {
-  run launcher_logs
-  echo "output = ${output}"
-  (( $status == 0 ))
-  [[ ${lines[4]} =~ run_final ]]
-}
-
-@test "kubectl delete of single-node wide-deep-large-ds-fp32-training-k8s" {
-  run $DETIK_CLIENT_NAME delete -f deployments/wide-deep-large-ds-fp32-training-k8s/single-node/user-mounted-nfs.yaml
-  echo "output = ${output}"
-  (( $status == 0 ))
-}
\ No newline at end of file
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/README.md b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/README.md
deleted file mode 100644
index 164a808ae..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/README.md
+++ /dev/null
@@ -1,365 +0,0 @@
-<!--- 0. Title -->
-# Wide and Deep Large Dataset FP32 training
-
-<!-- 10. Description -->
-
-This document has instructions for running [Wide and Deep](https://arxiv.org/pdf/1606.07792.pdf) FP32 training using
-Intel® Optimizations for TensorFlow* on Kubernetes*.
-
-
-
-<!--- 20. Download link -->
-## Download link
-
-[wide-deep-large-ds-fp32-training-k8s.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/wide-deep-large-ds-fp32-training-k8s.tar.gz)
-
-<!--- 30. Datasets -->
-## Dataset
-
-The large [Kaggle Display Advertising Challenge Dataset](https://www.kaggle.com/c/criteo-display-ad-challenge/data)
-will be used for training Wide and Deep. The
-[data](https://www.kaggle.com/c/criteo-display-ad-challenge/data) is from
-[Criteo](https://www.criteo.com) and has a field indicating if an ad was
-clicked (1) or not (0), along with integer and categorical features.
-
-Download large Kaggle Display Advertising Challenge Dataset from
-[Criteo Labs](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/).
-* Download the large version of train dataset from: https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/train.csv
-* Download the large version of evaluation dataset from: https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/eval.csv
-
-The directory where you've downloaded the `train.csv` and `eval.csv`
-files should be used as the `DATASET_DIR` when running [quickstart scripts](#quick-start-scripts).
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_training_check_accuracy.sh`](mlops/pipeline/user-mounted-nfs/train_and_serve.yaml#L50) | Trains the model for a specified number of steps (default is 500) and then compare the accuracy against the specified target accuracy. If the accuracy is not met, then script exits with error code 1. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L50) | Trains the model for 10 epochs if -- steps is not specified. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-
-These quickstart scripts are run in the following environment:
-* [Kubernetes](#kubernetes)
-
-<!--- 70. Kubernetes -->
-## Kubernetes
-
-Download and untar the model training package to get the yaml and config
-files for running training on a single node using Kubernetes.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/wide-deep-large-ds-fp32-training-k8s.tar.gz
-tar -xvf wide-deep-large-ds-fp32-training-k8s.tar.gz
-```
-
-### Execution
-
-The Kubernetes* package for `Wide and Deep Large Dataset FP32 training` includes single-node and pipeline kubernetes deployments.
-Within the single and pipeline deployments are common use cases that include storage and security variations that are common 
-across different kubernetes installations. The directory tree within the kubernetes package is shown below, where 
-single-node and pipeline directories are below the [mlops](https://en.wikipedia.org/wiki/MLOps) directory:
-Common use cases are found under the single and pipeline directories:
-
-```
-quickstart
-└── mlops
-    ├── pipeline
-    │       ├── user-allocated-pvc
-    │       └── user-mounted-nfs
-    └── single-node
-            ├── user-allocated-pvc
-            └── user-mounted-nfs
-```
-
-#### Prerequisites
-
-Both single-node and pipeline deployments use [kustomize-v3.8.7](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.7) to configure deployment parameters. Kustomize-v3.8.7 should be downloaded, extracted and moved to a directory within your PATH. You can verify that you've installed the correct version of kustomize by typing `kustomize version`. On MACOSX you would see:
-
-```
-{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:19:38Z GoOs:darwin GoArch:amd64}
-```
-
-#### Single-node Training
-
-Single node training is similar to the docker use case but is run within a pod.
-Training is done by submitting a pod.yaml to the k8s api-server which results in the pod creation and running
-the [`launch_benchmark.py`](mlops/single-node/user-mounted-nfs/pod.yaml#L50) script within the pod's container.
-
-In a terminal, `cd` to the single-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### User mounted nfs and user allocated pvc parameter values
-
-|     NAME   |                 VALUE    |        DESCRIPTION          |
-|------------|--------------------------|-----------------------------|
-| DATASET_DIR| /datasets                | input dataset directory     |
-| FS_ID      | 0                        | owner id of mounted volumes |
-| GROUP_ID   | 0                        | process group id            |
-| GROUP_NAME | root                     | process group name          |
-| NFS_PATH   | /nfs                     | nfs path                    |
-| NFS_SERVER | 0.0.0.0                  | nfs server                  |
-| PVC_NAME   | workdisk                 | model-builder | pvc name    |
-| PVC_PATH   | /pvc                     | model-builder | pvc path    |
-| USER_ID    | 0                        | process owner id            |
-| USER_NAME  | root                     | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the pod is deployed with the user's identity.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-##### Single-node training output
-
-Viewing the log output of the wide-deep-large-ds-fp32-training-k8s.tar.gz Pod is done by viewing the logs of the 
-training pod. This pod is found by filtering the list of pods for the name 'training'
-
-```
-kubectl get pods -oname|grep training|cut -c5-
-```
-
-This can be combined with the kubectl logs subcommand to tail the output of the training job
-
-```
-kubectl logs -f $(kubectl get pods -oname|grep training|cut -c5-)
-```
-
-##### Single-node training cleanup
-
-Removing the pod and related resources is done by running:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-#### Model Training and TF Serving Pipeline
-
-This pipeline runs the following steps using an Argo workflow:
-1. Train the model on a single node using the `fp32_training_check_accuracy.sh`
-   script. This script runs model training for a specified number of steps,
-   exports the saved model, and compares the accuacy against the value
-   specified in the `TARGET_ACCURACY` environment variable. If the model's
-   accuracy does not meet the target accuracy, this step will be retried
-   and continues training based on previous checkpoints in the specified
-   `CHECKPOINT_DIR`. If the `TARGET_ACCURACY` environment variable has
-   not been defined, then no accuracy check is done and it will continue
-   on to the next step, regardless of the model's accuracy.
-1. Deploy TensorFlow Serving containers with the saved model
-1. Create a service that exposes the TensorFlow Serving containers as a
-   NodePort
-
-The TensorFlow Serving steps in this pipeline follows the
-[TensorFlow Serving with Kubernetes instructions](https://www.tensorflow.org/tfx/serving/serving_kubernetes)
-with the exception that it does not use a Google Cloud Kubernetes
-cluster. Since the Kubernetes cluster being used does not have a load
-balancer, the configuration is setup for NodePort, which will allow
-external requests.
-In a terminal, `cd` to the multi-node directory. Each use case under this directory has parameters that can be changed 
-using kustomize's [cfg set](https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/commands/set.md)
-
-##### [Mlops](https://en.wikipedia.org/wiki/MLOps)
-
-###### User mounted nfs and user allocated pvc parameter values
-
-|     NAME        |                 VALUE    |        DESCRIPTION          |
-|-----------------|--------------------------|-----------------------------|
-| DATASET_DIR     | /datasets                | input dataset directory     |
-| FS_ID           | 0                        | owner id of mounted volumes |
-| GROUP_ID        | 0                        | process group id            |
-| GROUP_NAME      | root                     | process group name          |
-| NFS_PATH        | /nfs                     | nfs path                    |
-| NFS_SERVER      | 0.0.0.0                  | nfs server                  |
-| PVC_NAME        | workdisk                 | model-builder | pvc name    |
-| PVC_PATH        | /pvc                     | model-builder | pvc path    |
-| REPLICAS        | 3                        | replica number              |
-| RETRY_LIMIT     | 10                       | replica number              |
-| TARGET_ACCURACY | 0.74                     | target accuracy             |
-| TF_SERVING_PORT | 8501                     | tf serving port             |
-| USER_ID    | 0                             | process owner id            |
-| USER_NAME  | root                          | process owner name          |
-
-For the user mounted nfs use case, the user should change NFS_PATH and NFS_SERVER.
-
-For the user allocated pvc use case, the user should change PVC_NAME and PVC_PATH.
-
-For example to change the NFS_SERVER address the user would run:
-
-```
-kustomize cfg set . NFS_SERVER <ip address> -R
-```
-
-To change the PVC_NAME the user would run:
-
-```
-kustomize cfg set . PVC_NAME <PVC Name> -R
-```
-
-In both use cases, the user should change the values below so the argo workflow[^3] is deployed with the user's identity.
-
-[^3]: In order for the argo workflow to run as a non root user it must set the WorkflowExecutor to be k8sapi, otherwise the workflow will fail with "Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock". See argo issue [2239](https://github.com/argoproj/argo/issues/2239). Setting argo's WorkflowExecutor to k8sapi is described [here](https://argoproj.github.io/argo/workflow-executors/). This must be performed by devops.
-
-```
-kustomize cfg set . FS_ID <Group ID> -R
-kustomize cfg set . GROUP_ID <Group ID> -R
-kustomize cfg set . GROUP_NAME <Group Name> -R
-kustomize cfg set . USER_ID <User ID> -R
-kustomize cfg set . USER_NAME <User Name> -R
-```
-
-The user should change the default namespace of all the resources by running the kustomize command:
-
-```
-pushd <use-case>
-kustomize edit set namespace <User's namespace>
-popd
-```
-
-This will place all resources within the specified namespace. Note: this namespace should be created prior to deployment.
-
-The user can also change their default kubectl context by running
-
-```
-kubectl config set-context --current --namespace=<User's namespace>
-```
-
-Once the user has changed parameter values they can then deploy the use-case by running:
-
-```
-kustomize build  <use-case> > <use-case>.yaml
-kubectl apply -f <use-case>.yaml
-```
-
-Once the job has been submitted, the status and logs can be viewed using
-the Argo user inferface or from the command line using kubectl or argo.
-The commands below describe how to use kubectl to see the workflow, pods,
-and log files:
-
-```
-kubectl get wf
-kubectl get pods
-kubectl logs <pod name> main
-```
-
-##### TensorFlow Serving Client
-
-Once all the steps in the workflow have completed, the TensorFlow
-Serving GRPC client can be used to run inference on the served model.
-
-Prior to running the client script, install the following dependency in
-your environment:
-* tensorflow-serving-api
-
-The client script reads a csv file (in this example we are using the
-[eval.csv file](#dataset)), formats the data in for input parameter, and
-then calls the served model. Accuracy and benchmarking metrics are
-printed out.
-
-Run the [run_tf_serving_client.py](run_tf_serving_client.py) script with
-the `--help` flag to see the argument options:
-```
-$ python run_wide_deep_client.py --help
-usage: wide-deep-large-ds-fp32-training-k8s/quickstart/run_tf_serving_client.py [-h]
-       [-s SERVER] -d DATA_FILE [-b BATCH_SIZE] [-n NUM_ITERATION] [-w WARM_UP_ITERATION]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -s SERVER, --server SERVER
-                        Server URL and port (default=localhost:8500).
-  -d DATA_FILE, --data_file DATA_FILE
-                        Path to csv data file
-  -b BATCH_SIZE, --batch_size BATCH_SIZE
-                        Batch size to use (default=1).
-  -n NUM_ITERATION, --num_iteration NUM_ITERATION
-                        Number of times to repeat (default=40).
-  -w WARM_UP_ITERATION, --warm_up_iteration WARM_UP_ITERATION
-                        Number of initial iterations to ignore in benchmarking (default=10).
-```
-
-1. Find the `INTERNAL-IP` one of the nodes in your cluster using
-   `kubectl get nodes -o wide`. This IP should be used as the server URL
-   in the `--server` arg.
-
-1. Get the `NodePort` using `kubectl describe service`. This `NodePort`
-   should be used as the port in the `--server` arg.
-
-1. Run the client script with your preferred parameters. For example:
-   ```
-   python wide-deep-large-ds-fp32-training-k8s/quickstart/run_tf_serving_client.py -s <Internal IP>:<Node Port> -d <path to eval.csv> --b <batch size>
-   ```
-   The script will call the served model using data from the csv file
-   and output performance and accuracy metrics.
-
-##### Clean up the pipeline
-
-To clean up the model training/serving pipeline, delete the service,
-deployment, and other resources using the following commands:
-
-```
-kubectl delete -f <use-case>.yaml
-```
-
-<!--- 71. TroubleShooting -->
-## TroubleShooting
-
-- Pod doesn't start. Status is ErrImagePull.<br/>
-  Docker recently implemented rate limits.<br/>
-  See this [note](https://thenewstack.io/docker-hub-limits-what-they-are-and-how-to-route-around-them/) about rate limits and work-arounds.
-
-- Argo workflow steps do not execute.<br/>
-  Error from `argo get <workflow>` is 'failed to save outputs: Failed to establish pod watch: timed out waiting for the condition'.<br/>
-  See this argo [issue](https://github.com/argoproj/argo/issues/4186). This is due to the workflow running as non-root.<br/>
-  Devops will need to change the workflow-executor to k8sapi as described [here](https://github.com/argoproj/argo/blob/master/docs/workflow-executors.md).
-
-- MpiOperator can't create workers. Error is '/bin/sh: /etc/hosts: Permission denied'. This is due to a bug in mpi-operator in the 'latest' container image
-  when the workers run as non-root. See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-  Use the container images: mpioperator/mpi-operator:v02.3 and mpioperator/kubectl-delivery:v0.2.3.
-
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/Krmfile b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/Krmfile
deleted file mode 100644
index a795a2b5d..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,315 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint dir basename
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.DEPLOYMENT_NAME:
-      x-k8s-cli:
-        substitution:
-          name: DEPLOYMENT_NAME
-          pattern: ${MODEL_NAME}-deployment
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.FS_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: FS_ID_VALUE
-          pattern: "${FS_ID}"
-          values:
-          - marker: ${FS_ID}
-            ref: '#/definitions/io.k8s.cli.setters.FS_ID'
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/recommendation:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.substitutions.MODEL_CONFIGMAP_NAME:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_CONFIGMAP_NAME
-          pattern: mlops-scripts-${MODEL_NAME}
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/wide-deep-large-ds-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: wide-deep-large-ds-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training_check_accuracy.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.REPLICAS:
-      type: integer
-      description: replica number
-      x-k8s-cli:
-        setter:
-          name: REPLICAS
-          value: "3"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.REPLICAS_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: REPLICAS_VALUE
-          pattern: "${REPLICAS}"
-          values:
-          - marker: ${REPLICAS}
-            ref: '#/definitions/io.k8s.cli.setters.REPLICAS'
-    io.k8s.cli.setters.RETRY_LIMIT:
-      type: integer
-      description: replica number
-      x-k8s-cli:
-        setter:
-          name: RETRY_LIMIT
-          value: "10"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.SERVER_NAME:
-      x-k8s-cli:
-        substitution:
-          name: SERVER_NAME
-          pattern: ${MODEL_NAME}-server
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.substitutions.SERVICE_NAME:
-      x-k8s-cli:
-        substitution:
-          name: SERVICE_NAME
-          pattern: ${MODEL_NAME}-service
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.TARGET_ACCURACY:
-      type: number
-      description: target accuracy
-      x-k8s-cli:
-        setter:
-          name: TARGET_ACCURACY
-          value: "0.74"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.TARGET_ACCURACY_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: TARGET_ACCURACY_VALUE
-          pattern: "${TARGET_ACCURACY}"
-          values:
-          - marker: ${TARGET_ACCURACY}
-            ref: '#/definitions/io.k8s.cli.setters.TARGET_ACCURACY'
-    io.k8s.cli.setters.TF_SERVING_PORT:
-      type: integer
-      description: tf serving port
-      x-k8s-cli:
-        setter:
-          name: TF_SERVING_PORT
-          value: "8501"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.TF_SERVING_PORT_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: TF_SERVING_PORT_VALUE
-          pattern: "${TF_SERVING_PORT}"
-          values:
-          - marker: ${TF_SERVING_PORT}
-            ref: '#/definitions/io.k8s.cli.setters.TF_SERVING_PORT'
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index 59db797f5..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,17 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  CHECKPOINT_DIR: /pvc/root/wide-deep-large-ds-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/wide-deep-large-ds-fp32-training/models/recommendation/tensorflow/wide_deep_large_ds/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  TARGET_ACCURACY: "0.74" # {"$openapi":"TARGET_ACCURACY_VALUE"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index be32ca989..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- role.yaml
-- role-binding.yaml
-- train_and_serve.yaml
-namespace: default
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml
deleted file mode 100644
index 7137d310d..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role-binding.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-subjects:
-- kind: ServiceAccount
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role.yaml
deleted file mode 100644
index 7fa7aff84..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/role.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  - configmaps
-  - services
-  verbs:
-  - get
-  - list
-  - create
-  - update
-  - delete
-  - watch
-  - patch
-- apiGroups:
-  - ""
-  resources:
-  - pods/log
-  verbs:
-  - get
-  - watch
-- apiGroups:
-  - extensions
-  - apps
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 21aee480c..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/train_and_serve.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/train_and_serve.yaml
deleted file mode 100644
index 51dbbc8c3..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-allocated-pvc/train_and_serve.yaml
+++ /dev/null
@@ -1,145 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Workflow
-metadata:
-  name: wide-deep-large-ds-fp32-train-serve-wf # {"$openapi":"WORKFLOW_NAME"}
-spec:
-  entrypoint: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  serviceAccountName: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  arguments:
-    parameters:
-    - name: DEPLOYMENT_NAME
-      value: wide-deep-large-ds-fp32-training-deployment # {"$openapi":"DEPLOYMENT_NAME"}
-    - name: SERVER_NAME
-      value: wide-deep-large-ds-fp32-training-server # {"$openapi":"SERVER_NAME"}
-    - name: SERVICE_NAME
-      value: wide-deep-large-ds-fp32-training-service # {"$openapi":"SERVICE_NAME"}
-    - name: USER_ID
-      value: "0" # {"$openapi":"USER_ID_VALUE"}
-    - name: GROUP_ID
-      value: "0" # {"$openapi":"GROUP_ID_VALUE"}
-    - name: FS_ID
-      value: "0" # {"$openapi":"FS_ID_VALUE"}
-    - name: NFS_PATH
-      value: /nfs # {"$openapi":"NFS_PATH"}
-    - name: NFS_SERVER
-      value: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-    - name: OUTPUT_DIR
-      value: /pvc/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-    - name: REPLICAS
-      value: "3" # {"$openapi":"REPLICAS_VALUE"}
-    - name: TF_SERVING_PORT
-      value: "8501" # {"$openapi":"TF_SERVING_PORT_VALUE"}
-  templates:
-  - name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    steps:
-    - - name: train-wide-deep-large-ds
-        template: train-wide-deep-large-ds
-    - - name: wide-deep-large-ds-deployment
-        template: wide-deep-large-ds-deployment
-    - - name: wide-deep-large-ds-service
-        template: wide-deep-large-ds-service
-  - name: train-wide-deep-large-ds
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-      imagePullPolicy: IfNotPresent
-      workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-      command: # {"$openapi":"MODEL_COMMAND"}
-      - /workspace/wide-deep-large-ds-fp32-training/quickstart/fp32_training_check_accuracy.sh # {"$openapi":"MODEL_COMMAND"}
-      envFrom:
-      - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: datasets
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-      - name: nfs-path
-        mountPath: /nfs # {"$openapi":"NFS_PATH"}
-    retryStrategy:
-      limit: 10 # {"$openapi":"RETRY_LIMIT"}
-      retryPolicy: "OnFailure"
-    volumes:
-    - name: datasets
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: nfs-path
-      nfs:
-        server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-        path: /nfs # {"$openapi":"NFS_PATH"}
-  - name: wide-deep-large-ds-deployment
-    resource:
-      action: create
-      manifest: |
-        apiVersion: apps/v1
-        kind: Deployment
-        metadata:
-          name: {{workflow.parameters.DEPLOYMENT_NAME}}
-          ownerReferences:
-          - apiVersion: argoproj.io/v1alpha1
-            kind: Workflow
-            blockOwnerDeletion: true
-            name: "{{workflow.name}}"
-            uid: "{{workflow.uid}}"
-        spec:
-          selector:
-            matchLabels:
-              app: {{workflow.parameters.SERVER_NAME}}
-          replicas: {{workflow.parameters.REPLICAS}}
-          template:
-            metadata:
-              labels:
-                app: {{workflow.parameters.SERVER_NAME}}
-            spec:
-              securityContext:
-                runAsUser: {{workflow.parameters.USER_ID}}
-                runAsGroup: {{workflow.parameters.GROUP_ID}}
-                fsGroup: {{workflow.parameters.FS_ID}}
-              containers:
-              - name: {{workflow.parameters.SERVER_NAME}}
-                image: intel/intel-optimized-tensorflow-serving:latest
-                env:
-                - name: MODEL_BASE_PATH
-                  value: {{workflow.parameters.OUTPUT_DIR}}
-                - name: TF_CPP_MIN_VLOG_LEVEL
-                  value: "1"
-                volumeMounts:
-                - name: nfs-path
-                  mountPath: {{workflow.parameters.NFS_PATH}}
-                ports:
-                - containerPort: {{workflow.parameters.TF_SERVING_PORT}}
-              volumes:
-              - name: nfs-path
-                nfs:
-                  server: {{workflow.parameters.NFS_SERVER}}
-                  path: {{workflow.parameters.NFS_PATH}}
-  - name: wide-deep-large-ds-service
-    resource:
-      action: create
-      manifest: |
-        apiVersion: v1
-        kind: Service
-        metadata:
-          name: {{workflow.parameters.SERVICE_NAME}}
-          labels:
-            app: {{workflow.parameters.SERVER_NAME}}
-          ownerReferences:
-          - apiVersion: argoproj.io/v1alpha1
-            kind: Workflow
-            blockOwnerDeletion: true
-            name: "{{workflow.name}}"
-            uid: "{{workflow.uid}}"
-        spec:
-          # comment or delete the following line if you want to use a LoadBalancer
-          type: NodePort
-          # if your cluster supports it, uncomment the following to automatically create
-          # an external load-balanced IP for the frontend service.
-          # type: LoadBalancer
-          ports:
-          - protocol: TCP
-            port: 8501
-            targetPort: 8500
-          selector:
-            app: {{workflow.parameters.SERVER_NAME}}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/Krmfile b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/Krmfile
deleted file mode 100644
index f11c2a7ac..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,315 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint dir basename
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: checkpoints
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.DEPLOYMENT_NAME:
-      x-k8s-cli:
-        substitution:
-          name: DEPLOYMENT_NAME
-          pattern: ${MODEL_NAME}-deployment
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.FS_ID:
-      type: integer
-      description: owner id of mounted volumes
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.FS_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: FS_ID_VALUE
-          pattern: "${FS_ID}"
-          values:
-          - marker: ${FS_ID}
-            ref: '#/definitions/io.k8s.cli.setters.FS_ID'
-    io.k8s.cli.setters.GROUP_ID:
-      type: integer
-      description: process group id
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.IMAGE:
-      description: image name
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/recommendation:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.MODEL_COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.substitutions.MODEL_CONFIGMAP_NAME:
-      x-k8s-cli:
-        substitution:
-          name: MODEL_CONFIGMAP_NAME
-          pattern: mlops-scripts-${MODEL_NAME}
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/wide-deep-large-ds-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: wide-deep-large-ds-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training_check_accuracy.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir basename
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.REPLICAS:
-      type: integer
-      description: replica number
-      x-k8s-cli:
-        setter:
-          name: REPLICAS
-          value: "3"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.REPLICAS_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: REPLICAS_VALUE
-          pattern: "${REPLICAS}"
-          values:
-          - marker: ${REPLICAS}
-            ref: '#/definitions/io.k8s.cli.setters.REPLICAS'
-    io.k8s.cli.setters.RETRY_LIMIT:
-      type: integer
-      description: replica number
-      x-k8s-cli:
-        setter:
-          name: RETRY_LIMIT
-          value: "10"
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.SERVER_NAME:
-      x-k8s-cli:
-        substitution:
-          name: SERVER_NAME
-          pattern: ${MODEL_NAME}-server
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.substitutions.SERVICE_NAME:
-      x-k8s-cli:
-        substitution:
-          name: SERVICE_NAME
-          pattern: ${MODEL_NAME}-service
-          values:
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-    io.k8s.cli.setters.TARGET_ACCURACY:
-      type: number
-      description: target accuracy
-      x-k8s-cli:
-        setter:
-          name: TARGET_ACCURACY
-          value: "0.74"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.TARGET_ACCURACY_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: TARGET_ACCURACY_VALUE
-          pattern: "${TARGET_ACCURACY}"
-          values:
-          - marker: ${TARGET_ACCURACY}
-            ref: '#/definitions/io.k8s.cli.setters.TARGET_ACCURACY'
-    io.k8s.cli.setters.TF_SERVING_PORT:
-      type: integer
-      description: tf serving port
-      x-k8s-cli:
-        setter:
-          name: TF_SERVING_PORT
-          value: "8501"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.TF_SERVING_PORT_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: TF_SERVING_PORT_VALUE
-          pattern: "${TF_SERVING_PORT}"
-          values:
-          - marker: ${TF_SERVING_PORT}
-            ref: '#/definitions/io.k8s.cli.setters.TF_SERVING_PORT'
-    io.k8s.cli.setters.USER_ID:
-      type: integer
-      description: process owner id
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index 888cab333..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,17 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  CHECKPOINT_DIR: /nfs/root/wide-deep-large-ds-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  PYTHONPATH: /workspace/wide-deep-large-ds-fp32-training/models/recommendation/tensorflow/wide_deep_large_ds/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  TARGET_ACCURACY: "0.74" # {"$openapi":"TARGET_ACCURACY_VALUE"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index be32ca989..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- role.yaml
-- role-binding.yaml
-- train_and_serve.yaml
-namespace: default
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml
deleted file mode 100644
index 7137d310d..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role-binding.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-subjects:
-- kind: ServiceAccount
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role.yaml
deleted file mode 100644
index 7fa7aff84..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/role.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  - configmaps
-  - services
-  verbs:
-  - get
-  - list
-  - create
-  - update
-  - delete
-  - watch
-  - patch
-- apiGroups:
-  - ""
-  resources:
-  - pods/log
-  verbs:
-  - get
-  - watch
-- apiGroups:
-  - extensions
-  - apps
-  resources:
-  - deployments
-  verbs:
-  - get
-  - list
-  - watch
-  - create
-  - update
-  - patch
-  - delete
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 21aee480c..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/train_and_serve.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/train_and_serve.yaml
deleted file mode 100644
index ca100de0c..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/pipeline/user-mounted-nfs/train_and_serve.yaml
+++ /dev/null
@@ -1,145 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Workflow
-metadata:
-  name: wide-deep-large-ds-fp32-train-serve-wf # {"$openapi":"WORKFLOW_NAME"}
-spec:
-  entrypoint: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  serviceAccountName: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  arguments:
-    parameters:
-    - name: DEPLOYMENT_NAME
-      value: wide-deep-large-ds-fp32-training-deployment # {"$openapi":"DEPLOYMENT_NAME"}
-    - name: SERVER_NAME
-      value: wide-deep-large-ds-fp32-training-server # {"$openapi":"SERVER_NAME"}
-    - name: SERVICE_NAME
-      value: wide-deep-large-ds-fp32-training-service # {"$openapi":"SERVICE_NAME"}
-    - name: USER_ID
-      value: "0" # {"$openapi":"USER_ID_VALUE"}
-    - name: GROUP_ID
-      value: "0" # {"$openapi":"GROUP_ID_VALUE"}
-    - name: FS_ID
-      value: "0" # {"$openapi":"FS_ID_VALUE"}
-    - name: NFS_PATH
-      value: /nfs # {"$openapi":"NFS_PATH"}
-    - name: NFS_SERVER
-      value: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-    - name: OUTPUT_DIR
-      value: /nfs/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-    - name: REPLICAS
-      value: "3" # {"$openapi":"REPLICAS_VALUE"}
-    - name: TF_SERVING_PORT
-      value: "8501" # {"$openapi":"TF_SERVING_PORT_VALUE"}
-  templates:
-  - name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    steps:
-    - - name: train-wide-deep-large-ds
-        template: train-wide-deep-large-ds
-    - - name: wide-deep-large-ds-deployment
-        template: wide-deep-large-ds-deployment
-    - - name: wide-deep-large-ds-service
-        template: wide-deep-large-ds-service
-  - name: train-wide-deep-large-ds
-    securityContext:
-      runAsUser: 0 # {"$openapi":"USER_ID"}
-      runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-      fsGroup: 0 # {"$openapi":"FS_ID"}
-    container:
-      image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-      imagePullPolicy: IfNotPresent
-      workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-      command: # {"$openapi":"MODEL_COMMAND"}
-      - /workspace/wide-deep-large-ds-fp32-training/quickstart/fp32_training_check_accuracy.sh # {"$openapi":"MODEL_COMMAND"}
-      envFrom:
-      - configMapRef:
-          name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-      volumeMounts:
-      - name: datasets
-        mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-        readOnly: true
-      - name: nfs-path
-        mountPath: /nfs # {"$openapi":"NFS_PATH"}
-    retryStrategy:
-      limit: 10 # {"$openapi":"RETRY_LIMIT"}
-      retryPolicy: "OnFailure"
-    volumes:
-    - name: datasets
-      hostPath:
-        path: /datasets # {"$openapi":"DATASET_DIR"}
-    - name: nfs-path
-      nfs:
-        server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-        path: /nfs # {"$openapi":"NFS_PATH"}
-  - name: wide-deep-large-ds-deployment
-    resource:
-      action: create
-      manifest: |
-        apiVersion: apps/v1
-        kind: Deployment
-        metadata:
-          name: {{workflow.parameters.DEPLOYMENT_NAME}}
-          ownerReferences:
-          - apiVersion: argoproj.io/v1alpha1
-            kind: Workflow
-            blockOwnerDeletion: true
-            name: "{{workflow.name}}"
-            uid: "{{workflow.uid}}"
-        spec:
-          selector:
-            matchLabels:
-              app: {{workflow.parameters.SERVER_NAME}}
-          replicas: {{workflow.parameters.REPLICAS}}
-          template:
-            metadata:
-              labels:
-                app: {{workflow.parameters.SERVER_NAME}}
-            spec:
-              securityContext:
-                runAsUser: {{workflow.parameters.USER_ID}}
-                runAsGroup: {{workflow.parameters.GROUP_ID}}
-                fsGroup: {{workflow.parameters.FS_ID}}
-              containers:
-              - name: {{workflow.parameters.SERVER_NAME}}
-                image: intel/intel-optimized-tensorflow-serving:latest
-                env:
-                - name: MODEL_BASE_PATH
-                  value: {{workflow.parameters.OUTPUT_DIR}}
-                - name: TF_CPP_MIN_VLOG_LEVEL
-                  value: "1"
-                volumeMounts:
-                - name: nfs-path
-                  mountPath: {{workflow.parameters.NFS_PATH}}
-                ports:
-                - containerPort: {{workflow.parameters.TF_SERVING_PORT}}
-              volumes:
-              - name: nfs-path
-                nfs:
-                  server: {{workflow.parameters.NFS_SERVER}}
-                  path: {{workflow.parameters.NFS_PATH}}
-  - name: wide-deep-large-ds-service
-    resource:
-      action: create
-      manifest: |
-        apiVersion: v1
-        kind: Service
-        metadata:
-          name: {{workflow.parameters.SERVICE_NAME}}
-          labels:
-            app: {{workflow.parameters.SERVER_NAME}}
-          ownerReferences:
-          - apiVersion: argoproj.io/v1alpha1
-            kind: Workflow
-            blockOwnerDeletion: true
-            name: "{{workflow.name}}"
-            uid: "{{workflow.uid}}"
-        spec:
-          # comment or delete the following line if you want to use a LoadBalancer
-          type: NodePort
-          # if your cluster supports it, uncomment the following to automatically create
-          # an external load-balanced IP for the frontend service.
-          # type: LoadBalancer
-          ports:
-          - protocol: TCP
-            port: 8501
-            targetPort: 8500
-          selector:
-            app: {{workflow.parameters.SERVER_NAME}}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
deleted file mode 100644
index 4282da60b..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/Krmfile
+++ /dev/null
@@ -1,230 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: wide-deep-large-ds-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir base name
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint dir base name
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: checkpoints
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_ID:
-      description: process owner id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      description: filesystem id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/recommendation:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/wide-deep-large-ds-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${PVC_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${PVC_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.PVC_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
-    io.k8s.cli.setters.PVC_NAME:
-      description: pvc name
-      x-k8s-cli:
-        setter:
-          name: PVC_NAME
-          value: workdisk
-          isSet: true
-          required: true
-          setBy: model-builder
-    io.k8s.cli.setters.PVC_PATH:
-      description: pvc path
-      x-k8s-cli:
-        setter:
-          name: PVC_PATH
-          value: /pvc
-          isSet: true
-          required: true
-          setBy: model-builder
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
deleted file mode 100644
index 09d7c6156..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/config-map.yaml
+++ /dev/null
@@ -1,16 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /pvc/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  CHECKPOINT_DIR: /pvc/root/wide-deep-large-ds-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  PYTHONPATH: /workspace/wide-deep-large-ds-fp32-training/models/recommendation/tensorflow/wide_deep_large_ds/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
deleted file mode 100644
index 928de28c2..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/pod.yaml
+++ /dev/null
@@ -1,75 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: create-output-dir
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(OUTPUT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  - name: create-checkpoint-dir
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/wide-deep-large-ds-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=wide_deep_large_ds
-    - --precision=fp32
-    - --mode=training
-    - --framework=tensorflow
-    - --batch-size=512
-    - --data-location=$(DATASET_DIR)
-    - --output-dir=$(OUTPUT_DIR)
-    - --checkpoint=$(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: pvc-path
-      mountPath: /pvc # {"$openapi":"PVC_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: pvc-path
-    persistentVolumeClaim:
-      claimName: workdisk # {"$openapi":"PVC_NAME"}
-  restartPolicy: OnFailure
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
deleted file mode 100644
index 21aee480c..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-allocated-pvc/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
deleted file mode 100644
index fdd5cdf06..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/Krmfile
+++ /dev/null
@@ -1,212 +0,0 @@
-apiVersion: config.k8s.io/v1alpha1
-kind: Krmfile
-openAPI:
-  definitions:
-    io.k8s.cli.setters.MODEL_NAME:
-      description: name use-case
-      x-k8s-cli:
-        setter:
-          name: MODEL_NAME
-          value: wide-deep-large-ds-fp32-training
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.OUTPUT_DIR:
-      description: output dir base name
-      x-k8s-cli:
-        setter:
-          name: OUTPUT_DIR
-          value: output
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.CHECKPOINT_DIR:
-      description: checkpoint dir base name
-      x-k8s-cli:
-        setter:
-          name: CHECKPOINT_DIR
-          value: checkpoints
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.DATASET_DIR:
-      description: input dataset directory
-      x-k8s-cli:
-        setter:
-          name: DATASET_DIR
-          value: /datasets
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_ID:
-      description: process owner id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: USER_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.GROUP_ID:
-      description: process group id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: GROUP_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.FS_ID:
-      description: filesystem id
-      type: integer
-      x-k8s-cli:
-        setter:
-          name: FS_ID
-          value: "0"
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.REGISTRY:
-      description: image location
-      x-k8s-cli:
-        setter:
-          name: REGISTRY
-          value: docker.io
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.IMAGE:
-      x-k8s-cli:
-        substitution:
-          name: IMAGE
-          pattern: ${REGISTRY}/intel/recommendation:${IMAGE_VERSION}-${MODEL_NAME}${IMAGE_SUFFIX}
-          values:
-          - marker: ${REGISTRY}
-            ref: '#/definitions/io.k8s.cli.setters.REGISTRY'
-          - marker: ${IMAGE_VERSION}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_VERSION'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${IMAGE_SUFFIX}
-            ref: '#/definitions/io.k8s.cli.setters.IMAGE_SUFFIX'
-    io.k8s.cli.setters.IMAGE_VERSION:
-      description: image version
-      x-k8s-cli:
-        setter:
-          name: IMAGE_VERSION
-          value: tf-latest
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.substitutions.GROUP_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: GROUP_ID_VALUE
-          pattern: "${GROUP_ID}"
-          values:
-          - marker: ${GROUP_ID}
-            ref: '#/definitions/io.k8s.cli.setters.GROUP_ID'
-    io.k8s.cli.substitutions.USER_ID_VALUE:
-      x-k8s-cli:
-        substitution:
-          name: USER_ID_VALUE
-          pattern: "${USER_ID}"
-          values:
-          - marker: ${USER_ID}
-            ref: '#/definitions/io.k8s.cli.setters.USER_ID'
-    io.k8s.cli.setters.MODEL_DIR:
-      description: container model directory
-      x-k8s-cli:
-        setter:
-          name: MODEL_DIR
-          value: /workspace/wide-deep-large-ds-fp32-training
-          setBy: model-builder
-    io.k8s.cli.setters.MODEL_SCRIPT:
-      description: model script name
-      x-k8s-cli:
-        setter:
-          name: MODEL_SCRIPT
-          value: fp32_training.sh
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.NFS_PATH:
-      description: nfs path
-      x-k8s-cli:
-        setter:
-          name: NFS_PATH
-          value: /nfs
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.NFS_SERVER:
-      description: nfs server
-      x-k8s-cli:
-        setter:
-          name: NFS_SERVER
-          value: 0.0.0.0
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.COMMAND:
-      x-k8s-cli:
-        substitution:
-          name: COMMAND
-          pattern: ${MODEL_DIR}/quickstart/${MODEL_SCRIPT}
-          values:
-          - marker: ${MODEL_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_DIR'
-          - marker: ${MODEL_SCRIPT}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_SCRIPT'
-    io.k8s.cli.setters.IMAGE_SUFFIX:
-      description: appended to image name
-      x-k8s-cli:
-        setter:
-          name: IMAGE_SUFFIX
-          value: ""
-          isSet: true
-          setBy: model-builder
-    io.k8s.cli.setters.GROUP_NAME:
-      description: process group name
-      x-k8s-cli:
-        setter:
-          name: GROUP_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.setters.USER_NAME:
-      description: process owner name
-      x-k8s-cli:
-        setter:
-          name: USER_NAME
-          value: root
-          isSet: true
-          setBy: model-builder
-          required: true
-    io.k8s.cli.substitutions.OUTPUT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: OUTPUT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${OUTPUT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${OUTPUT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.OUTPUT_DIR'
-    io.k8s.cli.substitutions.CHECKPOINT_PATH:
-      x-k8s-cli:
-        substitution:
-          name: CHECKPOINT_PATH
-          pattern: ${NFS_PATH}/${USER_NAME}/${MODEL_NAME}/${CHECKPOINT_DIR}
-          values:
-          - marker: ${NFS_PATH}
-            ref: '#/definitions/io.k8s.cli.setters.NFS_PATH'
-          - marker: ${USER_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.USER_NAME'
-          - marker: ${MODEL_NAME}
-            ref: '#/definitions/io.k8s.cli.setters.MODEL_NAME'
-          - marker: ${CHECKPOINT_DIR}
-            ref: '#/definitions/io.k8s.cli.setters.CHECKPOINT_DIR'
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
deleted file mode 100644
index d1d2f862e..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/config-map.yaml
+++ /dev/null
@@ -1,16 +0,0 @@
-kind: ConfigMap
-apiVersion: v1
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-data:
-  DATASET_DIR: /datasets # {"$openapi":"DATASET_DIR"}
-  FS_ID: "0" # {"$openapi":"FS_ID_VALUE"}
-  GROUP_ID: "0" # {"$openapi":"GROUP_ID_VALUE"}
-  GROUP_NAME: root # {"$openapi":"GROUP_NAME"}
-  LD_LIBRARY_PATH: /usr/local/lib:/usr/local/lib/mpirun:$LD_LIBRARY_PATH
-  MODEL_DIR: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"} 
-  OUTPUT_DIR: /nfs/root/wide-deep-large-ds-fp32-training/output # {"$openapi":"OUTPUT_PATH"}
-  CHECKPOINT_DIR: /nfs/root/wide-deep-large-ds-fp32-training/checkpoints # {"$openapi":"CHECKPOINT_PATH"}
-  PYTHONPATH: /workspace/wide-deep-large-ds-fp32-training/models/recommendation/tensorflow/wide_deep_large_ds/training:$PYTHONPATH # {"$openapi":"PYTHONPATH"}
-  USER_ID: "0" # {"$openapi":"USER_ID_VALUE"}
-  USER_NAME: root # {"$openapi":"USER_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
deleted file mode 100644
index d17d12762..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/kustomization.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-- service-account.yaml
-- pod.yaml
-namespace: default
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
deleted file mode 100644
index a5846ba81..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/pod.yaml
+++ /dev/null
@@ -1,76 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-spec:
-  serviceAccountName: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-  securityContext:
-    runAsUser: 0 # {"$openapi":"USER_ID"}
-    runAsGroup: 0 # {"$openapi":"GROUP_ID"}
-    fsGroup: 0 # {"$openapi":"FS_ID"}
-  initContainers:
-  - name: create-output-dir
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(OUTPUT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  - name: create-checkpoint-dir
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - mkdir
-    args: 
-    - -p
-    - $(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  containers:
-  - name: single-node
-    image: docker.io/intel/recommendation:tf-latest-wide-deep-large-ds-fp32-training # {"$openapi":"IMAGE"}
-    imagePullPolicy: IfNotPresent
-    workingDir: /workspace/wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_DIR"}
-    command:
-    - python
-    args: # {"$openapi":"COMMAND"}
-    - /workspace/wide-deep-large-ds-fp32-training/benchmarks/launch_benchmark.py # {"$openapi":"COMMAND"}
-    - --model-name=wide_deep_large_ds
-    - --precision=fp32
-    - --mode=training
-    - --framework=tensorflow
-    - --batch-size=512
-    - --data-location=$(DATASET_DIR)
-    - --output-dir=$(OUTPUT_DIR)
-    - --checkpoint=$(CHECKPOINT_DIR)
-    envFrom:
-    - configMapRef:
-        name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
-    volumeMounts:
-    - name: datasets
-      mountPath: /datasets # {"$openapi":"DATASET_DIR"}
-      readOnly: true
-    - name: nfs-path
-      mountPath: /nfs # {"$openapi":"NFS_PATH"}
-  volumes:
-  - name: datasets
-    hostPath:
-      path: /datasets # {"$openapi":"DATASET_DIR"}
-  - name: nfs-path
-    nfs:
-      server: 0.0.0.0 # {"$openapi":"NFS_SERVER"}
-      path: /nfs # {"$openapi":"NFS_PATH"}
-  restartPolicy: OnFailure
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
deleted file mode 100644
index 21aee480c..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/mlops/single-node/user-mounted-nfs/service-account.yaml
+++ /dev/null
@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: wide-deep-large-ds-fp32-training # {"$openapi":"MODEL_NAME"}
diff --git a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/run_tf_serving_client.py b/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/run_tf_serving_client.py
deleted file mode 100644
index 4b85e116d..000000000
--- a/k8s/recommendation/tensorflow/wide_deep_large_ds/training/fp32/run_tf_serving_client.py
+++ /dev/null
@@ -1,228 +0,0 @@
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Usage: python wide_deep_benchmark.py -d <path-to-tfrecords-data-file>
-
-
-from __future__ import print_function
-
-import os
-import time
-import collections
-import argparse
-import grpc
-import numpy as np
-import tensorflow as tf
-
-from tensorflow_serving.apis import predict_pb2
-from tensorflow_serving.apis import prediction_service_pb2_grpc
-
-tf.compat.v1.disable_eager_execution()
-
-
-def check_for_link(value):
-    """
-    Throws an error if the specified path is a link. os.islink returns
-    True for sym links.  For files, we also look at the number of links in
-    os.stat() to determine if it's a hard link.
-    """
-    if os.path.islink(value) or \
-            (os.path.isfile(value) and os.stat(value).st_nlink > 1):
-        raise argparse.ArgumentTypeError("{} cannot be a link.".format(value))
-
-
-def check_valid_file(value):
-    """verifies filename exists and isn't a link"""
-    if value is not None:
-        if not os.path.isfile(value):
-            raise argparse.ArgumentTypeError(
-                "{} does not exist or is not a file.".
-                format(value))
-        check_for_link(value)
-    return value
-
-
-def generate_input_fn(filename, batch_size, num_epochs):
-    """
-    From train.py
-    """
-    CONTINUOUS_COLUMNS = ["I" + str(i) for i in range(1, 14)]  # 1-13 inclusive
-    CATEGORICAL_COLUMNS = ["C" + str(i) for i in
-                           range(1, 27)]  # 1-26 inclusive
-    LABEL_COLUMN = ["clicked"]
-    TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS
-
-    def parse_csv(value):
-        tf.compat.v1.logging.info('Parsing {}'.format(filename))
-        cont_defaults = [[0.0] for i in range(1, 14)]
-        cate_defaults = [[" "] for i in range(1, 27)]
-        label_defaults = [[0]]
-        column_headers = TRAIN_DATA_COLUMNS
-        record_defaults = label_defaults + cont_defaults + cate_defaults
-        columns = tf.io.decode_csv(value, record_defaults=record_defaults)
-        all_columns = collections.OrderedDict(zip(column_headers, columns))
-        labels = all_columns.pop(LABEL_COLUMN[0])
-        features = all_columns
-        return features, labels
-
-    # Extract lines from input files using the Dataset API.
-    dataset = tf.data.TextLineDataset(filename)
-    dataset = dataset.shuffle(buffer_size=20000)
-    dataset = dataset.repeat(num_epochs)
-    dataset = dataset.prefetch(batch_size)
-    dataset = dataset.batch(batch_size)
-    dataset = dataset.map(parse_csv, num_parallel_calls=28)
-    dataset = dataset.prefetch(1)
-    return dataset
-
-
-def input_fn(data_file, num_epochs, shuffle, batch_size):
-    """Generate an input function for the Estimator."""
-
-    def _parse_function(proto):
-        numeric_feature_names = ["numeric_1"]
-        string_feature_names = ["string_1"]
-        full_features_names = numeric_feature_names + string_feature_names + [
-            "label"]
-        feature_datatypes = [tf.io.FixedLenSequenceFeature([], tf.float32,
-                                                           default_value=0.0,
-                                                           allow_missing=True)] + [
-                                tf.io.FixedLenSequenceFeature(
-                                    [], tf.int64, default_value=0,
-                                    allow_missing=True)] + [
-                                tf.io.FixedLenSequenceFeature([], tf.int64,
-                                                              default_value=0,
-                                                              allow_missing=True)]
-        f = collections.OrderedDict(
-            zip(full_features_names, feature_datatypes))
-        parsed_features = tf.io.parse_example(proto, f)
-        parsed_feature_vals_num = [tf.reshape(
-            parsed_features["numeric_1"], shape=[-1, 13])]
-        parsed_feature_vals_str = [tf.reshape(
-            parsed_features["string_1"], shape=[-1, 2]) for i in
-            string_feature_names]
-        parsed_feature_vals = parsed_feature_vals_num + parsed_feature_vals_str
-        parsed_feature_vals_label = [tf.reshape(parsed_features[i], shape=[-1])
-                                     for i in ["label"]]
-        parsed_feature_vals = parsed_feature_vals + parsed_feature_vals_label
-        return parsed_feature_vals
-
-    # Extract lines from input files using the Dataset API.
-    dataset = tf.data.TFRecordDataset([data_file])
-    if shuffle:
-        dataset = dataset.shuffle(buffer_size=20000)
-    dataset = dataset.batch(batch_size)
-    dataset = dataset.map(_parse_function, num_parallel_calls=16)
-    return dataset
-
-
-def benchmark(batch_size=1, num_iteration=20, warm_up_iteration=10):
-    channel = grpc.insecure_channel(SERVER_URL)
-    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
-    request = predict_pb2.PredictRequest()
-    request.model_spec.name = 'wide-deep-large-ds-fp32-training'
-    request.model_spec.signature_name = 'predict'
-
-    i = 0
-    total_time = 0
-    res_dataset = generate_input_fn(DATA_FILE, batch_size=batch_size, num_epochs=1)
-    #res_dataset = input_fn(DATA_FILE, 1, False, batch_size)
-
-    CONTINUOUS_COLUMNS = ["I" + str(i) for i in range(1, 14)]  # 1-13 inclusive
-    CATEGORICAL_COLUMNS = ["C" + str(i) for i in range(1, 27)]  # 1-26 inclusive
-    LABEL_COLUMN = ["clicked"]
-    TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS
-    FEATURE_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS
-
-    iterator = tf.compat.v1.data.make_one_shot_iterator(res_dataset)
-    next_element = iterator.get_next()
-    with tf.compat.v1.Session() as sess:
-        sess.run(tf.compat.v1.global_variables_initializer())
-        try:
-            for _ in range(num_iteration):
-                correct_predictions = 0
-                i += 1
-                input_data = sess.run(next_element)
-
-                for c in FEATURE_COLUMNS:
-                    request.inputs[c].CopyFrom(
-                        tf.make_tensor_proto(input_data[0][c], shape=[1 * batch_size]))
-
-                # request.inputs['numeric'].CopyFrom(
-                #     tf.make_tensor_proto(input_data[0],
-                #                          shape=[1 * batch_size, 13]))
-                # request.inputs['categorical'].CopyFrom(
-                #     tf.make_tensor_proto(input_data[1],
-                #                          shape=[26 * batch_size, 2]))
-
-
-                start_time = time.time()
-                result = stub.Predict(request)
-                # print("-" * 20 + " result " + "-" * 20)
-                # print(result.outputs['probabilities'])
-                # print("-" * 20 + " result " + "-" * 20)
-                time_consume = time.time() - start_time
-                if i > warm_up_iteration:
-                    total_time += time_consume
-                for j in range(batch_size):
-                    if input_data[1][j] == np.argmax(
-                            result.outputs['probabilities'].float_val[
-                            j * 2:(j * 2) + 2]):
-                        correct_predictions += 1
-                print('Iteration %d: %.3f sec (accuracy: %.2f%%)' % (
-                i, time_consume, 100.0 * correct_predictions / batch_size))
-
-        except tf.errors.OutOfRangeError:
-            pass
-
-    time_average = total_time / (num_iteration - warm_up_iteration)
-    print('Average time: %.3f sec' % (time_average))
-    print('Batch size = %d' % batch_size)
-    if batch_size == 1:
-        print('Latency: %.3f ms' % (time_average * 1000))
-    print('Throughput: %.3f examples/sec' % (batch_size / time_average))
-
-
-if __name__ == '__main__':
-    ap = argparse.ArgumentParser()
-    ap.add_argument("-s", "--server", type=str, default='localhost:8500',
-                    help="Server URL and port (default=localhost:8500).")
-    ap.add_argument("-d", "--data_file", type=check_valid_file, required=True,
-                    help="Path to csv data file")
-    ap.add_argument("-b", "--batch_size", required=False, type=int, default=1,
-                    help="Batch size to use (default=1).")
-    ap.add_argument("-n", "--num_iteration", required=False, type=int,
-                    default=40,
-                    help="Number of times to repeat (default=40).")
-    ap.add_argument("-w", "--warm_up_iteration", required=False, type=int,
-                    default=10,
-                    help="Number of initial iterations to ignore in "
-                         "benchmarking (default=10).")
-
-    args = vars(ap.parse_args())
-
-    SERVER_URL = args['server']
-    DATA_FILE = args['data_file']
-    BATCH_SIZE = args['batch_size']
-    NUM_ITERATION = args['num_iteration']
-    WARM_UP_ITERATION = args['warm_up_iteration']
-
-    print('\n SERVER_URL: {} \n DATA_FILE: {}'.format(SERVER_URL, DATA_FILE))
-
-    print('\nStarting Wide and Deep model benchmarking for Latency with '
-          'batch_size={}, num_iteration=20, warm_up_iteration=10'.format(
-        BATCH_SIZE))
-    benchmark(batch_size=BATCH_SIZE, num_iteration=NUM_ITERATION,
-              warm_up_iteration=WARM_UP_ITERATION)
diff --git a/models/image_recognition/pytorch/common/main.py b/models/image_recognition/pytorch/common/main.py
index 87860b367..9f3f1ce6c 100644
--- a/models/image_recognition/pytorch/common/main.py
+++ b/models/image_recognition/pytorch/common/main.py
@@ -279,7 +279,8 @@ def main_worker(gpu, ngpus_per_node, args):
             else:
                 model = models.__dict__[args.arch]()
 
-
+    global num_classes
+    num_classes = model.fc.out_features
     if args.ipex:
         import intel_extension_for_pytorch as ipex
     # for ipex path, always convert model to channels_last for bf16, fp32.
@@ -726,7 +727,7 @@ def validate(val_loader, model, criterion, args):
                     images = torch.randn(args.batch_size, 3, 224, 224)
                     if args.ipex:
                         images = images.contiguous(memory_format=torch.channels_last)
-                    target = torch.arange(1, args.batch_size + 1).long()
+                    target = (torch.rand(args.batch_size) * num_classes).long()
                     if args.bf16:
                         images = images.to(torch.bfloat16)
                     if args.ipex and args.fp16:
diff --git a/models/image_recognition/pytorch/common/train.py b/models/image_recognition/pytorch/common/train.py
index 51ae9f851..b66d873cd 100644
--- a/models/image_recognition/pytorch/common/train.py
+++ b/models/image_recognition/pytorch/common/train.py
@@ -103,6 +103,8 @@
                     help='Initialize scale params in BN3 of a residual block to zeros instead ones. '
                          'Improves accuracy by 0.2~0.3 percent according to https://arxiv.org/abs/1706.02677'
                          'Used by Nvidia, but not part of MLPerf reference ')
+parser.add_argument('--warmup-iterations', default=-1, type=int, metavar='N',
+                    help='number of total warmup iterations to run')
 parser.add_argument('-i', '--iterations', default=-1, type=int, metavar='N',
                     help='number of total iterations to run')
 parser.add_argument('--train-no-eval', action='store_true', default=False,
@@ -164,7 +166,7 @@ def main_worker(args):
     # Optimizer
     if args.base_op.lower() == "lars":
         print("Creating LARS optimizer")
-        optimizer = create_optimizer_lars(model=model, lr=args.base_lr, epsilon=args.epsilon,
+        optimizer = ipex.optim._lars.create_optimizer_lars(model=model, lr=args.base_lr, epsilon=args.epsilon,
                                           momentum=args.momentum, weight_decay=args.weight_decay,
                                           bn_bias_separately=args.bn_bias_separately)
     else:                 
@@ -176,7 +178,7 @@ def main_worker(args):
     if args.ipex:
         print("using ipex to do training.....................")
         if args.bf16:
-            model, optimizer = ipex.optimize(model, dtype=torch.bfloat16, optimizer=optimizer)
+            model, optimizer = ipex.optimize(model, dtype=torch.bfloat16, optimizer=optimizer, split_master_weight_for_bf16=False)
         elif args.fp16:
             model, optimizer = ipex.optimize(model, dtype=torch.half, optimizer=optimizer)
         else:
@@ -268,6 +270,21 @@ def main_worker(args):
         scaler = torch.cpu.amp.GradScaler()
     else:
         scaler = None
+    # Train mode
+    model.train()
+
+    if args.ipex:
+        if args.bf16:
+            print("running ipex bfloat16 training step\n")
+        elif args.bf32:
+            print("running ipex bfloat32 training step\n")
+        elif args.fp16:
+            print("running ipex float16 training step\n")
+        else:
+            print("running ipex float32 training step\n")
+    
+    total_time = 0
+    compute_time = 0
 
     for epoch in range(args.start_epoch, args.epochs):
         
@@ -278,8 +295,90 @@ def main_worker(args):
             #TODO: add warmup for SGD path
             adjust_learning_rate(optimizer, epoch, args)
 
-        # train for one epoch
-        epoch_time = train(train_loader, model, criterion, optimizer, lr_scheduler, epoch, scaler, args)
+        # train for each epoch
+        
+        epoch_time = 0
+
+        batch_time = AverageMeter('Time', ':6.3f')      # Track total time = data-load + compute
+        data_time = AverageMeter('Data', ':6.3f')       # Track data-load time
+        losses = AverageMeter('Loss', ':.4e')
+        top1 = AverageMeter('Acc@1', ':6.2f')
+        top5 = AverageMeter('Acc@5', ':6.2f')
+        progress = ProgressMeter(
+            len(train_loader),
+            [batch_time, data_time, losses, top1, top5],
+            prefix="Epoch: [{}]".format(epoch))
+
+        start = time.time()
+        for i, (images, target) in enumerate(train_loader):
+
+            if args.distributed:
+                dist.barrier()
+
+            if args.ipex:
+                images = images.contiguous(memory_format=torch.channels_last)
+            
+            if i % args.print_freq == 0:
+                progress.display(i)
+            
+            #data loading time
+            t1 = time.time() - start
+            data_time.update(t1)
+
+            # Forward pass
+            if args.ipex and args.bf16:
+                with torch.cpu.amp.autocast():
+                    output = model(images)
+                output = output.to(torch.float32)
+            elif args.ipex and args.fp16:
+                with torch.cpu.amp.autocast(dtype=torch.half):
+                    output = model(images)
+                output = output.to(torch.float32)
+            else:
+                output = model(images)
+            loss = criterion(output, target)
+
+            # compute gradient and do optimizer step
+            optimizer.zero_grad()
+            if args.fp16:
+                scaler.scale(loss).backward()
+                if lr_scheduler:
+                    lr_scheduler.step()
+                scaler.step(optimizer)
+                scaler.update()
+            else:
+                loss.backward()
+                if lr_scheduler:
+                    lr_scheduler.step()
+                optimizer.step()
+
+            # measure elapsed time and reset timer
+            t2 = time.time() - start
+            batch_time.update(t2)
+            epoch_time += t2 - t1
+            if args.warmup_iterations < 0 or (epoch - args.start_epoch) * len(train_loader) + i + 1 > args.warmup_iterations:
+                total_time += t2
+                compute_time += t2 - t1
+
+            # measure accuracy and record loss
+            acc, counts = accuracy(output, target, topk=(1, 5))
+            acc1, acc5 = acc[0], acc[1]        
+            losses.update(loss.item(), images.size(0))
+            top1.update(acc1[0], images.size(0))
+            top5.update(acc5[0], images.size(0))   
+
+
+            start = time.time()
+            
+            if args.warmup_iterations > 0 and \
+            args.iterations > 0 and \
+            (epoch - args.start_epoch) * len(train_loader) + i + 1 >= args.iterations + args.warmup_iterations:
+                break
+            elif args.warmup_iterations < 0 and \
+            args.iterations > 0 and \
+            (epoch - args.start_epoch) * len(train_loader) + i + 1 >= args.iterations:
+                break
+            
         print("epoch: ", epoch, ", train_time(s): ", epoch_time)
         time_to_train += epoch_time
         print("time_to_train(s): ", time_to_train)
@@ -303,98 +402,36 @@ def main_worker(args):
 
             if best_acc1 > args.target_acc:
                 break
-
-    print("final time_to_train(s): ", time_to_train)
-
-def train(train_loader, model, criterion, optimizer, lr_scheduler, epoch, scaler, args):
-
-    epoch_time = 0
-
-    batch_time = AverageMeter('Time', ':6.3f')      # Track total time = data-load + compute
-    data_time = AverageMeter('Data', ':6.3f')       # Track data-load time
-    losses = AverageMeter('Loss', ':.4e')
-    top1 = AverageMeter('Acc@1', ':6.2f')
-    top5 = AverageMeter('Acc@5', ':6.2f')
-    progress = ProgressMeter(
-        len(train_loader),
-        [batch_time, data_time, losses, top1, top5],
-        prefix="Epoch: [{}]".format(epoch))
-
-    # Train mode
-    model.train()
-
-    if args.ipex:
-        if args.bf16:
-            print("running ipex bfloat16 training step\n")
-        elif args.bf32:
-            print("running ipex bfloat32 training step\n")
-        elif args.fp16:
-            print("running ipex float16 training step\n")
-        else:
-            print("running ipex float32 training step\n")
-
-    start = time.time()
-    for i, (images, target) in enumerate(train_loader):
-
-        if args.ipex:
-            images = images.contiguous(memory_format=torch.channels_last)
-
-        #data loading time
-        t1 = time.time() - start
-        data_time.update(t1)
-
-        # Forward pass
-        if args.ipex and args.bf16:
-            with torch.cpu.amp.autocast():
-                output = model(images)
-            output = output.to(torch.float32)
-        elif args.ipex and args.fp16:
-            with torch.cpu.amp.autocast(dtype=torch.half):
-                output = model(images)
-            output = output.to(torch.float32)
-        else:
-            output = model(images)
-        loss = criterion(output, target)
-
-        # compute gradient and do optimizer step
-        optimizer.zero_grad()
-        if args.fp16:
-            scaler.scale(loss).backward()
-            if lr_scheduler:
-                lr_scheduler.step()
-            scaler.step(optimizer)
-            scaler.update()
-        else:
-            loss.backward()
-            if lr_scheduler:
-                lr_scheduler.step()
-            optimizer.step()
-
-        # measure elapsed time and reset timer
-        t2 = time.time() - start
-        batch_time.update(t2)
-        epoch_time += t2 - t1
-
-        # measure accuracy and record loss
-        acc, counts = accuracy(output, target, topk=(1, 5))
-        acc1, acc5 = acc[0], acc[1]        
-        losses.update(loss.item(), images.size(0))
-        top1.update(acc1[0], images.size(0))
-        top5.update(acc5[0], images.size(0))   
-
-        if i % args.print_freq == 0:
-            progress.display(i)
-
-        if args.iterations > 0 and i + 1 == args.iterations:
+        
+        if args.warmup_iterations > 0 and \
+        args.iterations > 0 and \
+        (epoch - args.start_epoch) * len(train_loader) + i + 1 >= args.iterations + args.warmup_iterations:
             break
+        elif args.warmup_iterations < 0 and \
+        args.iterations > 0 and \
+        (epoch - args.start_epoch) * len(train_loader) + i + 1 >= args.iterations:
+            break
+    
+    # workaround oneccl bad termination issue.
+    if args.distributed:
+        dist.destroy_process_group()
 
-        start = time.time()
-
+    # Compute Throughput
     batch_size = args.local_batch_size
-    perf = batch_size / (batch_time.avg - data_time.avg)
-    print("Training throughput: {:.3f} fps".format(perf))
+    total_iterations = min((args.epochs - args.start_epoch) * len(train_loader), args.iterations) if args.iterations > 0 else (args.epochs - args.start_epoch) * len(train_loader)
+    if args.warmup_iterations > 0 and args.iterations < 0:
+        total_iterations -= args.warmup_iterations
+    if total_iterations <= 0:
+        print("Error! length of Train Data Loader or args.iterations are small than warmup iterations! Please put a larger args.iterations or a smaller args.warmup_iterations.")
+    perf = batch_size * total_iterations / compute_time
+    print(f"batch size is {batch_size}, total_iterantions is {total_iterations}, compute_time is {compute_time}")
+    print("Training throughput(compute): {:.3f} fps".format(perf))
+    perf2 = batch_size * total_iterations / total_time
+    print(f"batch size is {batch_size}, total_iterantions is {total_iterations}, total_time is {total_time}")
+    print("Training throughput(dataload+compute): {:.3f} fps".format(perf2))
+    
+    print("final time_to_train(s): ", time_to_train)
 
-    return epoch_time
 
 '''
 Distributed evaluation notes:
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference.py
index cf0312060..12ca8d32e 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference.py
+++ b/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference.py
@@ -105,6 +105,10 @@ def __init__(self):
       choices=['float32', 'fp16', 'bfloat16'],
       help='data type used for computation',
       default="float32")
+    arg_parser.add_argument(
+      '--onednn-graph', dest='onednn_graph',
+      help='enable OneDNN Graph',
+      action='store_true')
 
     self.args = arg_parser.parse_args()
     # validate the arguements
@@ -135,6 +139,8 @@ def run(self):
     infer_config.use_per_session_threads = 1
     infer_config.graph_options.rewrite_options.remapping = (
                   rewriter_config_pb2.RewriterConfig.AGGRESSIVE)
+    if self.args.onednn_graph:
+      infer_config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
     if dtype == 'fp16':
       infer_config.graph_options.rewrite_options.auto_mixed_precision = (
                     rewriter_config_pb2.RewriterConfig.ON)
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference_weight_sharing.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference_weight_sharing.py
index a5f6e27d2..1f7ea0d41 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference_weight_sharing.py
+++ b/models/image_recognition/tensorflow/resnet50v1_5/inference/cpu/eval_image_classifier_inference_weight_sharing.py
@@ -27,6 +27,7 @@
 from tensorflow.core.protobuf import rewriter_config_pb2
 
 import numpy as np
+import os
 import threading
 
 INPUTS = 'input_tensor'
@@ -98,6 +99,10 @@ def __init__(self):
       '--num-cores', dest='num_cores',
       help='number of cores',
       type=int, default=28)
+    arg_parser.add_argument(
+      '--onednn-graph', dest='onednn_graph',
+      help='enable OneDNN Graph',
+      action='store_true')
 
     self.args = arg_parser.parse_args()
     # validate the arguements
@@ -125,6 +130,8 @@ def run(self):
     infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
     infer_config.graph_options.rewrite_options.remapping = (
                   rewriter_config_pb2.RewriterConfig.AGGRESSIVE)
+    if self.args.onednn_graph:
+      infer_config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
 
     infer_graph = tf.Graph()
 
@@ -165,11 +172,21 @@ def run_model(data_sess, infer_sess, tid):
           if (num_images > warm_up):
             time_consume += end_time - start_time
           num_images+=1
-        throughput_list.append((steps-warm_up)/time_consume)
-        print('Instance num %d Avg Time/Iteration %f msec Throughput %f images/sec' %(tid, time_consume*1000/(steps - warm_up), (steps-warm_up)/time_consume))
+        if self.args.onednn_graph:
+          total_examples = (steps - warm_up) * self.args.batch_size
+          single_throughput = total_examples/time_consume
+          throughput_list.append(single_throughput)
+          print('Instance num %d Avg Time/Iteration %f msec Throughput %f images/sec' %(tid, time_consume*1000/total_examples, total_examples/time_consume))
+        else:
+          throughput_list.append((steps-warm_up)/time_consume)
+          print('Instance num %d Avg Time/Iteration %f msec Throughput %f images/sec' %(tid, time_consume*1000/(steps - warm_up), (steps-warm_up)/time_consume))
 
     if (not self.args.accuracy_only):
-      num_instances = self.args.num_intra_threads//4
+      if self.args.onednn_graph and os.environ.get("_ITEX_ONEDNN_GRAPH_COMPILER_BACKEND")=="1":
+        # graph compiler requires 1 core per instance
+        num_instances = self.args.num_intra_threads
+      else:
+        num_instances = self.args.num_intra_threads//4
       threads = []
       for i in range(1, num_instances+1):
         thread = threading.Thread(target=run_model, args=(data_sess,infer_sess1, i))
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/datasets.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/datasets.py
deleted file mode 100644
index 954a54d5c..000000000
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/datasets.py
+++ /dev/null
@@ -1,96 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Benchmark dataset utilities.
-"""
-
-from abc import abstractmethod
-import os
-
-import tensorflow as tf
-
-import preprocessing
-
-IMAGENET_NUM_TRAIN_IMAGES = 1281167
-IMAGENET_NUM_VAL_IMAGES = 50000
-IMAGENET_NUM_CLASSES = 1000
-
-class Dataset(object):
-  """Abstract class for cnn benchmarks dataset."""
-
-  def __init__(self, name, data_dir=None):
-    self.name = name
-    if data_dir is None:
-      raise ValueError('Data directory not specified')
-    self.data_dir = data_dir
-
-  def tf_record_pattern(self, subset):
-    return os.path.join(self.data_dir, '%s-*-of-*' % subset)
-
-  def reader(self):
-    return tf.compat.v1.TFRecordReader()
-
-  @abstractmethod
-  def num_classes(self):
-    pass
-
-  @abstractmethod
-  def num_examples_per_epoch(self, subset):
-    pass
-
-  def __str__(self):
-    return self.name
-
-
-class ImagenetData(Dataset):
-
-  def __init__(self, data_dir=None):
-    super(ImagenetData, self).__init__('ImageNet', data_dir)
-
-  def num_classes(self):
-    return IMAGENET_NUM_CLASSES
-
-  def num_examples_per_epoch(self, subset='train'):
-    if subset == 'train':
-      return IMAGENET_NUM_TRAIN_IMAGES
-    elif subset == 'validation':
-      return IMAGENET_NUM_VAL_IMAGES
-    elif subset == 'calibrate' or subset == 'calibration':
-      return 100
-    else:
-      raise ValueError('Invalid data subset "%s"' % subset)
-
-  def get_image_preprocessor(self):
-    return preprocessing.RecordInputImagePreprocessor
\ No newline at end of file
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/eval_image_classifier_inference.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/eval_image_classifier_inference.py
deleted file mode 100644
index c9b4e6be3..000000000
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/eval_image_classifier_inference.py
+++ /dev/null
@@ -1,339 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-import time
-import sys
-from argparse import ArgumentParser
-
-import tensorflow as tf
-#import tensorflow.tools.graph_transforms as graph_transforms
-from tensorflow.python.tools.optimize_for_inference_lib import optimize_for_inference
-from tensorflow.python.framework import dtypes
-
-import datasets as datasets
-import numpy as np
-from tensorflow.core.protobuf import rewriter_config_pb2 
-
-from tensorflow.python.client import timeline
-
-from optimize_for_benchmark import optimize_for_benchmark
-
-INPUTS = 'input_tensor'
-OUTPUTS = 'softmax_tensor'
-#INPUTS = 'import/input_tensor:0'
-#OUTPUTS = 'import/softmax_tensor:0'
-OPTIMIZATION = 'strip_unused_nodes remove_nodes(op=Identity, op=CheckNumerics) fold_constants(ignore_errors=true) fold_batch_norms fold_old_batch_norms'
-
-RESNET_IMAGE_SIZE = 224
-tf.random.set_seed(1234)
-np.random.seed(1234)
-
-class eval_classifier_optimized_graph:
-  """Evaluate image classifier with optimized TensorFlow graph"""
-
-  def __init__(self):
-
-    arg_parser = ArgumentParser(description='Parse args')
-
-    arg_parser.add_argument('-b', "--batch-size",
-                            help="Specify the batch size. If this " \
-                                 "parameter is not specified or is -1, the " \
-                                 "largest ideal batch size for the model will " \
-                                 "be used.",
-                            dest="batch_size", type=int, default=-1)
-
-    arg_parser.add_argument('-e', "--num-inter-threads",
-                            help='The number of inter-thread.',
-                            dest='num_inter_threads', type=int, default=1)
-
-    arg_parser.add_argument('-a', "--num-intra-threads",
-                            help='The number of intra-thread.',
-                            dest='num_intra_threads', type=int, default=0)
-
-    arg_parser.add_argument('-m', "--model-name",
-                            help='Specify the model name to run benchmark for',
-                            dest='model_name')
-
-    arg_parser.add_argument('-g', "--input-graph",
-                            help='Specify the input graph for the transform tool',
-                            dest='input_graph')
-
-    arg_parser.add_argument('-d', "--data-location",
-                            help='Specify the location of the data. '
-                                 'If this parameter is not specified, '
-                                 'the benchmark will use random/dummy data.',
-                            dest="data_location", default=None)
-
-    arg_parser.add_argument('-r', "--accuracy-only",
-                            help='For accuracy measurement only.',
-                            dest='accuracy_only', action='store_true')
-    arg_parser.add_argument('--calibrate', dest='calibrate',
-                            help='Run accuracy with calibration data,'
-                                 'to generate min_max ranges, calibrate=[True/False]',
-                            type=bool, default=False)
-    arg_parser.add_argument("--results-file-path",
-                            help="File path for the inference results",
-                            dest="results_file_path", default=None)
-    arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                            help="number of warmup steps")
-    arg_parser.add_argument("--steps", type=int, default=50,
-                            help="number of steps")
-
-    arg_parser.add_argument(
-      '--data-num-inter-threads', dest='data_num_inter_threads',
-      help='number threads across operators',
-      type=int, default=32)
-    arg_parser.add_argument(
-      '--data-num-intra-threads', dest='data_num_intra_threads',
-      help='number threads for data layer operator',
-      type=int, default=14)
-    arg_parser.add_argument(
-      '--num-cores', dest='num_cores',
-      help='number of cores',
-      type=int, default=28)
-
-    arg_parser.add_argument("--benchmark",
-                            help='Run in benchmark mode.',
-                            dest='benchmark', action='store_true')
-
-    self.args = arg_parser.parse_args()
-    # validate the arguements
-    self.validate_args()
-
-  def write_results_output(self, predictions, filenames, labels):
-    # If a results_file_path is provided, write the predictions to the file
-    if self.args.results_file_path:
-      top_predictions = np.argmax(tf.cast(predictions, tf.float32), 1)
-      with open(self.args.results_file_path, "a") as fp:
-        for filename, expected_label, top_prediction in zip(filenames, labels, top_predictions):
-          fp.write("{},{},{}\n".format(filename, expected_label, top_prediction))
-
-  def run(self):
-    """run benchmark with optimized graph"""
-
-    print("Run inference")
-
-    data_config = tf.compat.v1.ConfigProto()
-    data_config.intra_op_parallelism_threads = self.args.data_num_intra_threads
-    data_config.inter_op_parallelism_threads = self.args.data_num_inter_threads
-    data_config.use_per_session_threads = 1
-
-    #data_config.graph_options.rewrite_options.disable_meta_optimizer = rewriter_config_pb2.RewriterConfig.OFF
-    data_config.graph_options.rewrite_options.layout_optimizer = rewriter_config_pb2.RewriterConfig.OFF
-
-    infer_config = tf.compat.v1.ConfigProto()
-    infer_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    infer_config.use_per_session_threads = 1
-    #infer_config.graph_options.rewrite_options.disable_meta_optimizer = rewriter_config_pb2.RewriterConfig.OFF
-    infer_config.graph_options.rewrite_options.layout_optimizer = rewriter_config_pb2.RewriterConfig.OFF
-
-    data_graph = tf.Graph()
-    with data_graph.as_default():
-      if (self.args.data_location):
-        print("Inference with real data.")
-        if self.args.calibrate:
-            subset = 'calibration'
-        else:
-            subset = 'validation'
-        dataset = datasets.ImagenetData(self.args.data_location)
-        preprocessor = dataset.get_image_preprocessor()(
-            RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, self.args.batch_size,
-            num_cores=self.args.num_cores,
-            resize_method='crop')
-
-        images, labels, filenames = preprocessor.minibatch(dataset, subset=subset)
-
-        # If a results file path is provided, then start the prediction output file
-        if self.args.results_file_path:
-          with open(self.args.results_file_path, "w+") as fp:
-            fp.write("filename,actual,prediction\n")
-      else:
-        print("Inference with dummy data.")
-        input_shape = [self.args.batch_size, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, 3]
-        images = tf.random.uniform(input_shape, 0.0, 255.0, dtype=tf.float32, name='synthetic_images')
-        images = tf.cast(images, tf.qint8)
-
-    infer_graph = tf.Graph()
-    with infer_graph.as_default():
-      graph_def = tf.compat.v1.GraphDef()
-      with tf.compat.v1.gfile.FastGFile(self.args.input_graph, 'rb') as input_file:
-        input_graph_content = input_file.read()
-        graph_def.ParseFromString(input_graph_content)
-
-      #output_graph = graph_transforms.TransformGraph(graph_def,
-      #                                   [INPUTS], [OUTPUTS], [OPTIMIZATION])
-      output_graph = optimize_for_inference(graph_def, [INPUTS], 
-                                      [OUTPUTS], dtypes.qint8.as_datatype_enum, False)
-
-      if self.args.benchmark:
-        input_shape = [self.args.batch_size, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, 3]
-        _np_qint8 = np.dtype([("qint8", np.int8, 1)])
-        dummy_input = np.random.randn(*input_shape).astype(_np_qint8)
-        output_graph = optimize_for_benchmark(output_graph, dtypes.qint8.as_datatype_enum, dummy_input)
-
-      tf.import_graph_def(output_graph, name='')
-
-    # Definite input and output Tensors for detection_graph
-    input_tensor = infer_graph.get_tensor_by_name('input_tensor:0')
-    output_tensor = infer_graph.get_tensor_by_name('softmax_tensor:0')
-
-    data_sess  = tf.compat.v1.Session(graph=data_graph,  config=data_config)
-    infer_sess = tf.compat.v1.Session(graph=infer_graph, config=infer_config)
-
-    num_processed_images = 0
-    num_remaining_images = dataset.num_examples_per_epoch(subset=subset) - num_processed_images \
-        if self.args.data_location else datasets.IMAGENET_NUM_VAL_IMAGES
-
-    options = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
-    run_metadata = tf.compat.v1.RunMetadata()
-    do_profile = False
-    #do_profile = True
-    profile_idx = 20
-
-    if (not self.args.accuracy_only):
-      iteration = 0
-      warm_up_iteration = self.args.warmup_steps
-      total_run = self.args.steps
-      total_time = 0
-      al = 64*16
-      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
-      dtype = np.dtype(np.uint8)
-      nbytes = self.args.batch_size * 224 * 224 * 3 * dtype.itemsize
-      buf = np.empty(nbytes + al, dtype=np.uint8)
-      start_index = -buf.ctypes.data % al
-      x = buf[start_index:start_index + nbytes].view(_np_qint8).reshape(self.args.batch_size, 224, 224, 3)
-      x = np.asarray(x, dtype=_np_qint8)
-
-      while num_remaining_images >= self.args.batch_size and iteration < total_run:
-        iteration += 1
-        tf_filenames = None
-        np_labels = None
-        data_load_start = time.time()
-        if self.args.results_file_path:
-          image_np, np_labels, tf_filenames = data_sess.run([images, labels, filenames])
-        else:
-          image_np = data_sess.run(images)
-
-        data_load_time = time.time() - data_load_start
-
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-        image_np = np.asarray(image_np, dtype=_np_qint8)
-        for i in range(image_np.shape[0]):
-          x[i, 0, 0, 0] = image_np[i, 0, 0, 0]
-        t = image_np[0, 0, 0 , 0]
-        image_np[0, 0, 0 , 0] = 0
-        image_np[0, 0, 0 , 0] = t
-        image_np = x
-        image_np = np.asarray(image_np, dtype=_np_qint8)
-        sys.stdout.flush()
-        start_time = time.time()
-        if do_profile:
-          predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np}, options=options, run_metadata=run_metadata)
-        else:
-          if self.args.benchmark:
-            predictions = infer_sess.run(output_tensor)
-          else:
-            predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np})
-        time_consume = time.time() - start_time
-
-        # Write out the file name, expected label, and top prediction
-        self.write_results_output(predictions, tf_filenames, np_labels)
-
-        # only add data loading time for real data, not for dummy data
-        if self.args.data_location:
-          time_consume += data_load_time
-
-        if iteration == profile_idx and do_profile:
-          fetched_timeline = timeline.Timeline(run_metadata.step_stats)
-          chrome_trace = fetched_timeline.generate_chrome_trace_format()
-          with open('timeline_01.json', 'w') as f:
-            f.write(chrome_trace)
-
-        print('Iteration %d: %.6f sec' % (iteration, time_consume))
-        if iteration > warm_up_iteration:
-          total_time += time_consume
-        sys.stdout.flush()
-
-      time_average = total_time / (iteration - warm_up_iteration)
-      print('Average time: %.6f sec' % (time_average))
-
-      print('Batch size = %d' % self.args.batch_size)
-      if (self.args.batch_size == 1):
-        print('Latency: %.3f ms' % (time_average * 1000))
-      # print throughput for both batch size 1 and 128
-      print('Throughput: %.3f images/sec' % (self.args.batch_size / time_average))
-      sys.stdout.flush()
-
-    else: # accuracy check
-      total_accuracy1, total_accuracy5 = (0.0, 0.0)
-
-      while num_remaining_images >= self.args.batch_size:
-        # Reads and preprocess data
-        tf_filenames = None
-        if self.args.results_file_path:
-          np_images, np_labels, tf_filenames = data_sess.run([images, labels, filenames])
-        else:
-          np_images, np_labels = data_sess.run([images, labels])
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        start_time = time.time()
-        # Compute inference on the preprocessed data
-        predictions = infer_sess.run(output_tensor,
-                               {input_tensor: np_images})
-        elapsed_time = time.time() - start_time
-
-        # Write out the file name, expected label, and top prediction
-        self.write_results_output(predictions, tf_filenames, np_labels)
-
-        with tf.Graph().as_default() as accu_graph:
-          accuracy1 = tf.reduce_sum(
-            tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions, tf.float32),
-                                   targets=tf.constant(np_labels), k=1), tf.float32))
-
-          accuracy5 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions, dtype=tf.float32),
-                targets=tf.constant(np_labels), k=5), tf.float32))
-          with tf.compat.v1.Session() as accu_sess:
-            np_accuracy1, np_accuracy5 = accu_sess.run([accuracy1, accuracy5])
-
-          total_accuracy1 += np_accuracy1
-          total_accuracy5 += np_accuracy5
-
-        elapsed_time *= 1000
-        print("Iteration time: %0.4f ms" % elapsed_time)
-        print("Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
-                  % (num_processed_images, total_accuracy1 / num_processed_images,
-                     total_accuracy5 / num_processed_images))
-        sys.stdout.flush()
-
-  def validate_args(self):
-    """validate the arguments"""
-
-    if not self.args.data_location:
-      if self.args.accuracy_only:
-        raise ValueError("You must use real data for accuracy measurement.")
-
-
-if __name__ == "__main__":
-  evaluate_opt_graph = eval_classifier_optimized_graph()
-  evaluate_opt_graph.run()
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/optimize_for_benchmark.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/optimize_for_benchmark.py
deleted file mode 100644
index ff65baaa8..000000000
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/optimize_for_benchmark.py
+++ /dev/null
@@ -1,72 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-from tensorflow.core.framework import graph_pb2
-from tensorflow.core.framework import node_def_pb2
-from tensorflow.python.framework import tensor_util
-from tensorflow.python.tools.optimize_for_inference_lib import ensure_graph_is_valid
-
-import numpy as np
-
-def optimize_for_benchmark(input_graph_def, const_dtype, dummy_input):
-  ensure_graph_is_valid(input_graph_def)
-  optimized_graph_def = change_placehoder_to_const(input_graph_def, const_dtype, dummy_input)
-  ensure_graph_is_valid(input_graph_def)
-  return optimized_graph_def
-
-def change_placehoder_to_const(input_graph_def, const_dtype, dummy_input):
-  result_graph_def = graph_pb2.GraphDef()
-  for node in input_graph_def.node:
-    if node.op == 'Placeholder':
-      new_const = node_def_pb2.NodeDef()
-      new_const.op = 'Const'
-      new_const.name = node.name
-      new_const.attr["dtype"].CopyFrom(node.attr["dtype"])
-      tensor_proto = tensor_util.make_tensor_proto(dummy_input,
-                                                   const_dtype,
-                                                   dummy_input.shape)
-      new_const.attr["value"].tensor.CopyFrom(tensor_proto)
-      result_graph_def.node.extend([new_const])
-    else:
-      new_node = node_def_pb2.NodeDef()
-      new_node.CopyFrom(node)
-      retained_input = []
-      for input_node in new_node.input:
-        retained_input.append(input_node)
-      new_node.input[:] = retained_input
-
-      result_graph_def.node.extend([new_node])
-
-  return result_graph_def
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/preprocessing.py b/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/preprocessing.py
deleted file mode 100644
index 0228a1e36..000000000
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/preprocessing.py
+++ /dev/null
@@ -1,178 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import tensorflow as tf
-from tensorflow.python.data.experimental import parallel_interleave
-from tensorflow.python.data.experimental import map_and_batch
-from tensorflow.python.platform import gfile
-
-
-def parse_example_proto(example_serialized):
-  """Parses an Example proto containing a training example of an image.
-  """
-  # Dense features in Example proto.
-  feature_map = {
-    'image/encoded': tf.io.FixedLenFeature([], dtype=tf.string,
-                                        default_value=''),
-    'image/class/label': tf.io.FixedLenFeature([1], dtype=tf.int64,
-                                            default_value=-1),
-    'image/filename': tf.io.FixedLenFeature([], dtype=tf.string,
-                                         default_value="")
-  }
-  sparse_float32 = tf.io.VarLenFeature(dtype=tf.float32)
-  # Sparse features in Example proto.
-  feature_map.update(
-    {k: sparse_float32 for k in ['image/object/bbox/xmin',
-                                 'image/object/bbox/ymin',
-                                 'image/object/bbox/xmax',
-                                 'image/object/bbox/ymax']})
-
-  features = tf.io.parse_single_example(serialized=example_serialized, features=feature_map)
-  label = tf.cast(features['image/class/label'], dtype=tf.int32)
-  filename = tf.cast(features['image/filename'], dtype=tf.string)
-
-  return features['image/encoded'], label, filename
-
-
-def eval_image(image, height, width, resize_method,
-               central_fraction=0.875, scope=None):
-
-  with tf.compat.v1.name_scope('eval_image'):
-    if resize_method == 'crop':
-      shape = tf.shape(input=image)
-      image = tf.cond(pred=tf.less(shape[0], shape[1]),
-                      true_fn=lambda: tf.image.resize(image,
-                                                     tf.convert_to_tensor(value=[256, 256 * shape[1] / shape[0]],
-                                                                          dtype=tf.int32)),
-                      false_fn=lambda: tf.image.resize(image,
-                                                     tf.convert_to_tensor(value=[256 * shape[0] / shape[1], 256],
-                                                                          dtype=tf.int32)))
-
-      shape = tf.shape(input=image)
-      y0 = (shape[0] - height) // 2
-      x0 = (shape[1] - width) // 2
-      distorted_image = tf.image.crop_to_bounding_box(image, y0, x0, height, width)
-      distorted_image.set_shape([height, width, 3])
-      means = tf.broadcast_to([123.68, 116.78, 103.94], tf.shape(input=distorted_image))
-      return distorted_image - means
-    else:  # bilinear
-      if image.dtype != tf.float32:
-        image = tf.image.convert_image_dtype(image, dtype=tf.float32)
-      # Crop the central region of the image with an area containing 87.5% of
-      # the original image.
-      if central_fraction:
-        image = tf.image.central_crop(image, central_fraction=central_fraction)
-
-      if height and width:
-        # Resize the image to the specified height and width.
-        image = tf.expand_dims(image, 0)
-        image = tf.image.resize(image, [height, width],
-                                         method=tf.image.ResizeMethod.BILINEAR)
-        image = tf.squeeze(image, [0])
-      image = tf.subtract(image, 0.5)
-      image = tf.multiply(image, 2.0)
-      return image
-
-class RecordInputImagePreprocessor(object):
-  """Preprocessor for images with RecordInput format."""
-
-  def __init__(self,
-               height,
-               width,
-               batch_size,
-               num_cores,
-               resize_method="bilinear"):
-
-    self.height = height
-    self.width = width
-    self.batch_size = batch_size
-    self.num_cores = num_cores
-    self.resize_method = resize_method
-
-  def parse_and_preprocess(self, value):
-    # parse
-    image_buffer, label_index, filename = parse_example_proto(value)
-    # preprocess
-    image = tf.image.decode_jpeg(
-      image_buffer, channels=3, fancy_upscaling=False, dct_method='INTEGER_FAST')
-    image = eval_image(image, self.height, self.width, self.resize_method)
-    image, _ , _ = tf.quantization.quantize(image, -123.68, 151.06, tf.qint8)
-    return (image, label_index, filename)
-
-  def minibatch(self, dataset, subset, cache_data=False):
-
-    with tf.compat.v1.name_scope('batch_processing'):
-
-      glob_pattern = dataset.tf_record_pattern(subset)
-      file_names = gfile.Glob(glob_pattern)
-      if not file_names:
-        raise ValueError('Found no files in --data_dir matching: {}'
-                         .format(glob_pattern))
-      ds = tf.data.TFRecordDataset.list_files(file_names)
-
-      ds = ds.apply(
-        parallel_interleave(
-          tf.data.TFRecordDataset, cycle_length=self.num_cores, block_length=5,
-          sloppy=True,
-          buffer_output_elements=10000, prefetch_input_elements=10000))
-
-      if cache_data:
-        ds = ds.take(1).cache().repeat()
-
-      ds = ds.prefetch(buffer_size=10000)
-      #ds = ds.prefetch(buffer_size=self.batch_size)
-
-      # num of parallel batches not greater than 56
-      max_num_parallel_batches = min(56, 2 * self.num_cores)
-      ds = ds.apply(
-        map_and_batch(
-          map_func=self.parse_and_preprocess,
-          batch_size=self.batch_size,
-          num_parallel_batches=max_num_parallel_batches,
-          num_parallel_calls=None))
-
-      ds = ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
-
-      ds_iterator = tf.compat.v1.data.make_one_shot_iterator(ds)
-      images, labels, filename = ds_iterator.get_next()
-      # reshape
-      labels = tf.reshape(labels, [self.batch_size])
-      filename = tf.reshape(filename, [self.batch_size])
-
-      return images, labels, filename
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/imagenet_main.py b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/imagenet_main.py
index f67b1c7fd..f8917b16e 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/imagenet_main.py
+++ b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/imagenet_main.py
@@ -43,7 +43,6 @@
 
 import numpy.random
 import tensorflow as tf  # pylint: disable=g-bad-import-order
-tf.compat.v1.disable_eager_execution()
 
 from mlperf_compliance import mlperf_log
 from mlperf_resnet import imagenet_preprocessing
@@ -168,7 +167,6 @@ def parse_record(raw_record, is_training, dtype):
 
 
 def input_fn(is_training, data_dir, batch_size, num_epochs=1, num_gpus=None,
-             datasets_num_private_threads=None,
              dtype=tf.float32):
   """Input function which provides batches for train or eval.
 
@@ -194,11 +192,6 @@ def input_fn(is_training, data_dir, batch_size, num_epochs=1, num_gpus=None,
   # Convert to individual records
   dataset = dataset.flat_map(tf.data.TFRecordDataset)
 
-  if datasets_num_private_threads:
-    options = tf.data.Options()
-    options.experimental_threading.private_threadpool_size = datasets_num_private_threads
-    dataset = dataset.with_options(options)
-
   return resnet_run_loop.process_record_dataset(
       dataset=dataset,
       is_training=is_training,
@@ -212,9 +205,9 @@ def input_fn(is_training, data_dir, batch_size, num_epochs=1, num_gpus=None,
   )
 
 
-def get_synth_input_fn(use_bfloat16):
+def get_synth_input_fn():
   return resnet_run_loop.get_synth_input_fn(
-      _DEFAULT_IMAGE_SIZE, _DEFAULT_IMAGE_SIZE, _NUM_CHANNELS, _NUM_CLASSES, use_bfloat16)
+      _DEFAULT_IMAGE_SIZE, _DEFAULT_IMAGE_SIZE, _NUM_CHANNELS, _NUM_CLASSES)
 
 
 ###############################################################################
@@ -363,7 +356,7 @@ def main(argv):
                           value=_NUM_IMAGES['train'])
   mlperf_log.resnet_print(key=mlperf_log.PREPROC_NUM_EVAL_EXAMPLES,
                           value=_NUM_IMAGES['validation'])
-  input_function = flags.use_synthetic_data and get_synth_input_fn(flags.use_bfloat16) or input_fn
+  input_function = flags.use_synthetic_data and get_synth_input_fn() or input_fn
 
   resnet_run_loop.resnet_main(seed,
       flags, imagenet_model_fn, input_function,
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_model.py b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_model.py
index c23d8fb1e..68922ceb8 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_model.py
+++ b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_model.py
@@ -189,7 +189,7 @@ def _bottleneck_block_v1(inputs, filters, training, projection_shortcut,
   mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_SHORTCUT_ADD)
   # TODO(nhasabni): temporarily replacing Add by AddN for performance.
   # Remove it later once we optimize this in graph.
-  inputs = tf.math.add(inputs, shortcut)
+  inputs = tf.math.add_n([inputs, shortcut])
 
   mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_RELU)
   inputs = tf.nn.relu(inputs)
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_run_loop.py b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_run_loop.py
index 2e8404b20..fabe67eb7 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_run_loop.py
+++ b/models/image_recognition/tensorflow/resnet50v1_5/training/cpu/mlperf_resnet/resnet_run_loop.py
@@ -123,7 +123,8 @@ def process_record_dataset(dataset, is_training, batch_size, shuffle_buffer,
       tf.data.experimental.map_and_batch(
           lambda value: parse_record_fn(value, is_training, dtype),
           batch_size=batch_size,
-          num_parallel_batches=1))
+          num_parallel_batches=1,
+          drop_remainder=True))
 
   # Operations between the final prefetch and the get_next call to the iterator
   # will happen synchronously during run time. We prefetch here again to
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/eval_image_classifier_inference.py b/models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/eval_image_classifier_inference.py
deleted file mode 100644
index 833d0981f..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/eval_image_classifier_inference.py
+++ /dev/null
@@ -1,237 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-import time
-from argparse import ArgumentParser
-
-import tensorflow as tf
-from tensorflow.python.tools.optimize_for_inference_lib import optimize_for_inference
-from tensorflow.python.framework import dtypes
-from tensorflow.core.protobuf import saved_model_pb2
-from tensorflow.core.protobuf import rewriter_config_pb2
-from tensorflow.python.platform import tf_logging
-
-from inference.preprocess import datasets
-import numpy as np
-
-INPUTS = 'pixel_values'
-OUTPUTS = 'Identity'
-
-RESNET_IMAGE_SIZE = 224
-IMAGENET_VALIDATION_IMAGES = 50000
-
-class eval_classifier_optimized_graph:
-  """Evaluate image classifier with optimized TensorFlow graph"""
-
-  def __init__(self):
-
-    arg_parser = ArgumentParser(description='Parse args')
-
-    arg_parser.add_argument('-b', "--batch-size",
-                            help="Specify the batch size. If this " \
-                                 "parameter is not specified or is -1, the " \
-                                 "largest ideal batch size for the model will " \
-                                 "be used.",
-                            dest="batch_size", type=int, default=-1)
-
-    arg_parser.add_argument('-e', "--num-inter-threads",
-                            help='The number of inter-thread.',
-                            dest='num_inter_threads', type=int, default=0)
-
-    arg_parser.add_argument('-a', "--num-intra-threads",
-                            help='The number of intra-thread.',
-                            dest='num_intra_threads', type=int, default=0)
-
-    arg_parser.add_argument('-m', "--model-name",
-                            help='Specify the model name to run benchmark for',
-                            dest='model_name')
-
-    arg_parser.add_argument('-g', "--input-graph",
-                            help='Specify the input graph for the transform tool',
-                            dest='input_graph')
-
-    arg_parser.add_argument('-d', "--data-location",
-                            help='Specify the location of the data. '
-                                 'If this parameter is not specified, '
-                                 'the benchmark will use random/dummy data.',
-                            dest="data_location", default=None)
-
-    arg_parser.add_argument('-r', "--accuracy-only",
-                            help='For accuracy measurement only.',
-                            dest='accuracy_only', action='store_true')
-
-    arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                            help="number of warmup steps")
-    arg_parser.add_argument("--steps", type=int, default=50,
-                            help="number of steps")
-    arg_parser.add_argument(
-        '--data-num-inter-threads', dest='data_num_inter_threads',
-        help='number threads across operators',
-        type=int, default=16)
-    arg_parser.add_argument(
-        '--data-num-intra-threads', dest='data_num_intra_threads',
-        help='number threads for data layer operator',
-        type=int, default=14)
-
-    # parse the arguments
-    self.args = arg_parser.parse_args()
-    # validate the arguements
-    self.validate_args()
-
-  def run(self):
-    """run benchmark with optimized graph"""
-
-    print("Run inference in BF16 precision.")
-
-    data_config = tf.compat.v1.ConfigProto()
-    data_config.intra_op_parallelism_threads = self.args.data_num_intra_threads
-    data_config.inter_op_parallelism_threads = self.args.data_num_inter_threads
-    data_config.use_per_session_threads = 1
-
-    infer_config = tf.compat.v1.ConfigProto()
-    infer_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    infer_config.use_per_session_threads = 1
-    infer_config.graph_options.rewrite_options.auto_mixed_precision_onednn_bfloat16 = rewriter_config_pb2.RewriterConfig.ON
-
-    data_graph = tf.Graph()
-    with data_graph.as_default():
-      if (self.args.data_location):
-        print("Inference with real data.")
-        dataset = datasets.ImagenetData(self.args.data_location)
-        preprocessor = dataset.get_image_preprocessor()(
-            RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, self.args.batch_size,
-            intra_threads=self.args.num_intra_threads,
-            resize_method='crop')
-
-        images, labels = preprocessor.minibatch(dataset, subset='validation')
-        images = tf.transpose(a=images, perm=[0, 3, 1, 2])
-
-      else:
-        print("Inference with dummy data.")
-        input_shape = [self.args.batch_size, 3, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE]
-        images = tf.random.uniform(input_shape, 0.0, 255.0, dtype=tf.float32, name='synthetic_images')
-
-
-    # Load the frozen model
-    sm = saved_model_pb2.SavedModel()
-    with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-        sm.ParseFromString(f.read())
-    g_def = sm.meta_graphs[0].graph_def
-    with tf.Graph().as_default() as infer_graph:
-        tf.import_graph_def(g_def, name='')
-
-
-    # Definite input and output Tensors for detection_graph
-    input_tensor = infer_graph.get_tensor_by_name('pixel_values:0')
-    output_tensor = infer_graph.get_tensor_by_name('Identity:0')
-
-    data_sess = tf.compat.v1.Session(graph=data_graph,  config=data_config)
-    infer_sess = tf.compat.v1.Session(graph=infer_graph, config=infer_config)
-
-    num_processed_images = 0
-    num_remaining_images = IMAGENET_VALIDATION_IMAGES
-
-    if (not self.args.accuracy_only):
-      print("Benchmark")
-      iteration = 0
-      warm_up_iteration = self.args.warmup_steps
-      total_run = self.args.steps
-      total_time = 0
-
-      while num_remaining_images >= self.args.batch_size and iteration < total_run:
-        iteration += 1
-
-        # Reads and preprocess data
-        data_load_start = time.time()
-        image_np = data_sess.run(images)
-        data_load_time = time.time() - data_load_start
-
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        tf_logging.warn('---> Start iteration{0}'.format(str(iteration)))
-        start_time = time.time()
-        predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np})
-        time_consume = time.time() - start_time
-        tf_logging.warn('---> Stop iteration{0}'.format(str(iteration)))
-
-        # only add data loading time for real data, not for dummy data
-        if self.args.data_location:
-          time_consume += data_load_time
-
-        print('Iteration %d: %.6f sec' % (iteration, time_consume))
-        if iteration > warm_up_iteration:
-          total_time += time_consume
-
-      time_average = total_time / (iteration - warm_up_iteration)
-      print('Average time: %.6f sec' % (time_average))
-
-      print('Batch size = %d' % self.args.batch_size)
-      if (self.args.batch_size == 1):
-        print('Latency: %.3f ms' % (time_average * 1000))
-      # print throughput for both batch size 1 and 128
-      print('Throughput: %.3f images/sec' % (self.args.batch_size / time_average))
-
-    else: # accuracy check
-      print("Accuracy Check")
-      total_accuracy1, total_accuracy5 = (0.0, 0.0)
-
-      while num_remaining_images >= self.args.batch_size:
-        # Reads and preprocess data
-        np_images, np_labels = data_sess.run([images, labels])
-        np_labels -= 1
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        start_time = time.time()
-        # Compute inference on the preprocessed data
-        predictions = infer_sess.run(output_tensor,
-                               {input_tensor: np_images})
-        elapsed_time = time.time() - start_time
-        with tf.Graph().as_default() as accu_graph:
-          # Putting all code within this make things faster.
-          accuracy1 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=1), tf.float32))
-
-          accuracy5 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=5), tf.float32))
-          with tf.compat.v1.Session() as accu_sess:
-            np_accuracy1, np_accuracy5 = accu_sess.run([accuracy1, accuracy5])
-          total_accuracy1 += np_accuracy1
-          total_accuracy5 += np_accuracy5
-          print("Iteration time: %0.4f ms" % elapsed_time)
-          print("Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
-              % (num_processed_images, total_accuracy1 / num_processed_images,
-                 total_accuracy5 / num_processed_images))
-
-  def validate_args(self):
-    """validate the arguments"""
-
-    if not self.args.data_location:
-      if self.args.accuracy_only:
-        raise ValueError("You must use real data for accuracy measurement.")
-
-
-if __name__ == "__main__":
-  evaluate_opt_graph = eval_classifier_optimized_graph()
-  evaluate_opt_graph.run()
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/fp16/eval_image_classifier_inference.py b/models/image_recognition/tensorflow/vision_transformer/inference/fp16/eval_image_classifier_inference.py
deleted file mode 100644
index e6f9b4248..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/inference/fp16/eval_image_classifier_inference.py
+++ /dev/null
@@ -1,237 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-import time
-from argparse import ArgumentParser
-
-import tensorflow as tf
-from tensorflow.python.tools.optimize_for_inference_lib import optimize_for_inference
-from tensorflow.python.framework import dtypes
-from tensorflow.core.protobuf import saved_model_pb2
-from tensorflow.core.protobuf import rewriter_config_pb2
-from tensorflow.python.platform import tf_logging
-
-from inference.preprocess import datasets
-import numpy as np
-
-INPUTS = 'pixel_values'
-OUTPUTS = 'Identity'
-
-RESNET_IMAGE_SIZE = 224
-IMAGENET_VALIDATION_IMAGES = 50000
-
-class eval_classifier_optimized_graph:
-  """Evaluate image classifier with optimized TensorFlow graph"""
-
-  def __init__(self):
-
-    arg_parser = ArgumentParser(description='Parse args')
-
-    arg_parser.add_argument('-b', "--batch-size",
-                            help="Specify the batch size. If this " \
-                                 "parameter is not specified or is -1, the " \
-                                 "largest ideal batch size for the model will " \
-                                 "be used.",
-                            dest="batch_size", type=int, default=-1)
-
-    arg_parser.add_argument('-e', "--num-inter-threads",
-                            help='The number of inter-thread.',
-                            dest='num_inter_threads', type=int, default=0)
-
-    arg_parser.add_argument('-a', "--num-intra-threads",
-                            help='The number of intra-thread.',
-                            dest='num_intra_threads', type=int, default=0)
-
-    arg_parser.add_argument('-m', "--model-name",
-                            help='Specify the model name to run benchmark for',
-                            dest='model_name')
-
-    arg_parser.add_argument('-g', "--input-graph",
-                            help='Specify the input graph for the transform tool',
-                            dest='input_graph')
-
-    arg_parser.add_argument('-d', "--data-location",
-                            help='Specify the location of the data. '
-                                 'If this parameter is not specified, '
-                                 'the benchmark will use random/dummy data.',
-                            dest="data_location", default=None)
-
-    arg_parser.add_argument('-r', "--accuracy-only",
-                            help='For accuracy measurement only.',
-                            dest='accuracy_only', action='store_true')
-
-    arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                            help="number of warmup steps")
-    arg_parser.add_argument("--steps", type=int, default=50,
-                            help="number of steps")
-    arg_parser.add_argument(
-        '--data-num-inter-threads', dest='data_num_inter_threads',
-        help='number threads across operators',
-        type=int, default=16)
-    arg_parser.add_argument(
-        '--data-num-intra-threads', dest='data_num_intra_threads',
-        help='number threads for data layer operator',
-        type=int, default=14)
-
-    # parse the arguments
-    self.args = arg_parser.parse_args()
-    # validate the arguements
-    self.validate_args()
-
-  def run(self):
-    """run benchmark with optimized graph"""
-
-    print("Run inference in FP16 precision.")
-
-    data_config = tf.compat.v1.ConfigProto()
-    data_config.intra_op_parallelism_threads = self.args.data_num_intra_threads
-    data_config.inter_op_parallelism_threads = self.args.data_num_inter_threads
-    data_config.use_per_session_threads = 1
-
-    infer_config = tf.compat.v1.ConfigProto()
-    infer_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    infer_config.use_per_session_threads = 1
-    infer_config.graph_options.rewrite_options.auto_mixed_precision = rewriter_config_pb2.RewriterConfig.ON
-
-    data_graph = tf.Graph()
-    with data_graph.as_default():
-      if (self.args.data_location):
-        print("Inference with real data.")
-        dataset = datasets.ImagenetData(self.args.data_location)
-        preprocessor = dataset.get_image_preprocessor()(
-            RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, self.args.batch_size,
-            intra_threads=self.args.num_intra_threads,
-            resize_method='crop')
-
-        images, labels = preprocessor.minibatch(dataset, subset='validation')
-        images = tf.transpose(a=images, perm=[0, 3, 1, 2])
-
-      else:
-        print("Inference with dummy data.")
-        input_shape = [self.args.batch_size, 3, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE]
-        images = tf.random.uniform(input_shape, 0.0, 255.0, dtype=tf.float32, name='synthetic_images')
-
-
-    # Load the frozen model
-    sm = saved_model_pb2.SavedModel()
-    with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-        sm.ParseFromString(f.read())
-    g_def = sm.meta_graphs[0].graph_def
-    with tf.Graph().as_default() as infer_graph:
-        tf.import_graph_def(g_def, name='')
-
-
-    # Definite input and output Tensors for detection_graph
-    input_tensor = infer_graph.get_tensor_by_name('pixel_values:0')
-    output_tensor = infer_graph.get_tensor_by_name('Identity:0')
-
-    data_sess = tf.compat.v1.Session(graph=data_graph,  config=data_config)
-    infer_sess = tf.compat.v1.Session(graph=infer_graph, config=infer_config)
-
-    num_processed_images = 0
-    num_remaining_images = IMAGENET_VALIDATION_IMAGES
-
-    if (not self.args.accuracy_only):
-      print("Benchmark")
-      iteration = 0
-      warm_up_iteration = self.args.warmup_steps
-      total_run = self.args.steps
-      total_time = 0
-
-      while num_remaining_images >= self.args.batch_size and iteration < total_run:
-        iteration += 1
-
-        # Reads and preprocess data
-        data_load_start = time.time()
-        image_np = data_sess.run(images)
-        data_load_time = time.time() - data_load_start
-
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        tf_logging.warn('---> Start iteration{0}'.format(str(iteration)))
-        start_time = time.time()
-        predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np})
-        time_consume = time.time() - start_time
-        tf_logging.warn('---> Stop iteration{0}'.format(str(iteration)))
-
-        # only add data loading time for real data, not for dummy data
-        if self.args.data_location:
-          time_consume += data_load_time
-
-        print('Iteration %d: %.6f sec' % (iteration, time_consume))
-        if iteration > warm_up_iteration:
-          total_time += time_consume
-
-      time_average = total_time / (iteration - warm_up_iteration)
-      print('Average time: %.6f sec' % (time_average))
-
-      print('Batch size = %d' % self.args.batch_size)
-      if (self.args.batch_size == 1):
-        print('Latency: %.3f ms' % (time_average * 1000))
-      # print throughput for both batch size 1 and 128
-      print('Throughput: %.3f images/sec' % (self.args.batch_size / time_average))
-
-    else: # accuracy check
-      print("Accuracy Check")
-      total_accuracy1, total_accuracy5 = (0.0, 0.0)
-
-      while num_remaining_images >= self.args.batch_size:
-        # Reads and preprocess data
-        np_images, np_labels = data_sess.run([images, labels])
-        np_labels -= 1
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        start_time = time.time()
-        # Compute inference on the preprocessed data
-        predictions = infer_sess.run(output_tensor,
-                               {input_tensor: np_images})
-        elapsed_time = time.time() - start_time
-        with tf.Graph().as_default() as accu_graph:
-          # Putting all code within this make things faster.
-          accuracy1 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=1), tf.float32))
-
-          accuracy5 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=5), tf.float32))
-          with tf.compat.v1.Session() as accu_sess:
-            np_accuracy1, np_accuracy5 = accu_sess.run([accuracy1, accuracy5])
-          total_accuracy1 += np_accuracy1
-          total_accuracy5 += np_accuracy5
-          print("Iteration time: %0.4f ms" % elapsed_time)
-          print("Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
-              % (num_processed_images, total_accuracy1 / num_processed_images,
-                 total_accuracy5 / num_processed_images))
-
-  def validate_args(self):
-    """validate the arguments"""
-
-    if not self.args.data_location:
-      if self.args.accuracy_only:
-        raise ValueError("You must use real data for accuracy measurement.")
-
-
-if __name__ == "__main__":
-  evaluate_opt_graph = eval_classifier_optimized_graph()
-  evaluate_opt_graph.run()
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/fp32/eval_image_classifier_inference.py b/models/image_recognition/tensorflow/vision_transformer/inference/fp32/eval_image_classifier_inference.py
deleted file mode 100644
index 777ead09d..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/inference/fp32/eval_image_classifier_inference.py
+++ /dev/null
@@ -1,235 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-import time
-from argparse import ArgumentParser
-
-import tensorflow as tf
-from tensorflow.python.tools.optimize_for_inference_lib import optimize_for_inference
-from tensorflow.python.framework import dtypes
-from tensorflow.core.protobuf import saved_model_pb2
-from tensorflow.python.platform import tf_logging
-
-from inference.preprocess import datasets
-import numpy as np
-
-INPUTS = 'pixel_values'
-OUTPUTS = 'Identity'
-
-RESNET_IMAGE_SIZE = 224
-IMAGENET_VALIDATION_IMAGES = 50000
-
-class eval_classifier_optimized_graph:
-  """Evaluate image classifier with optimized TensorFlow graph"""
-
-  def __init__(self):
-
-    arg_parser = ArgumentParser(description='Parse args')
-
-    arg_parser.add_argument('-b', "--batch-size",
-                            help="Specify the batch size. If this " \
-                                 "parameter is not specified or is -1, the " \
-                                 "largest ideal batch size for the model will " \
-                                 "be used.",
-                            dest="batch_size", type=int, default=-1)
-
-    arg_parser.add_argument('-e', "--num-inter-threads",
-                            help='The number of inter-thread.',
-                            dest='num_inter_threads', type=int, default=0)
-
-    arg_parser.add_argument('-a', "--num-intra-threads",
-                            help='The number of intra-thread.',
-                            dest='num_intra_threads', type=int, default=0)
-
-    arg_parser.add_argument('-m', "--model-name",
-                            help='Specify the model name to run benchmark for',
-                            dest='model_name')
-
-    arg_parser.add_argument('-g', "--input-graph",
-                            help='Specify the input graph for the transform tool',
-                            dest='input_graph')
-
-    arg_parser.add_argument('-d', "--data-location",
-                            help='Specify the location of the data. '
-                                 'If this parameter is not specified, '
-                                 'the benchmark will use random/dummy data.',
-                            dest="data_location", default=None)
-
-    arg_parser.add_argument('-r', "--accuracy-only",
-                            help='For accuracy measurement only.',
-                            dest='accuracy_only', action='store_true')
-
-    arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                            help="number of warmup steps")
-    arg_parser.add_argument("--steps", type=int, default=50,
-                            help="number of steps")
-    arg_parser.add_argument(
-        '--data-num-inter-threads', dest='data_num_inter_threads',
-        help='number threads across operators',
-        type=int, default=16)
-    arg_parser.add_argument(
-        '--data-num-intra-threads', dest='data_num_intra_threads',
-        help='number threads for data layer operator',
-        type=int, default=14)
-
-    # parse the arguments
-    self.args = arg_parser.parse_args()
-    # validate the arguements
-    self.validate_args()
-
-  def run(self):
-    """run benchmark with optimized graph"""
-
-    print("Run inference in FP32 precision.")
-
-    data_config = tf.compat.v1.ConfigProto()
-    data_config.intra_op_parallelism_threads = self.args.data_num_intra_threads
-    data_config.inter_op_parallelism_threads = self.args.data_num_inter_threads
-    data_config.use_per_session_threads = 1
-
-    infer_config = tf.compat.v1.ConfigProto()
-    infer_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    infer_config.use_per_session_threads = 1
-
-    data_graph = tf.Graph()
-    with data_graph.as_default():
-      if (self.args.data_location):
-        print("Inference with real data.")
-        dataset = datasets.ImagenetData(self.args.data_location)
-        preprocessor = dataset.get_image_preprocessor()(
-            RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, self.args.batch_size,
-            intra_threads=self.args.num_intra_threads,
-            resize_method='crop')
-
-        images, labels = preprocessor.minibatch(dataset, subset='validation')
-        images = tf.transpose(a=images, perm=[0, 3, 1, 2])
-
-      else:
-        print("Inference with dummy data.")
-        input_shape = [self.args.batch_size, 3, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE]
-        images = tf.random.uniform(input_shape, 0.0, 255.0, dtype=tf.float32, name='synthetic_images')
-
-
-    # Load the frozen model
-    sm = saved_model_pb2.SavedModel()
-    with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-        sm.ParseFromString(f.read())
-    g_def = sm.meta_graphs[0].graph_def
-    with tf.Graph().as_default() as infer_graph:
-        tf.import_graph_def(g_def, name='')
-
-
-    # Definite input and output Tensors for detection_graph
-    input_tensor = infer_graph.get_tensor_by_name('pixel_values:0')
-    output_tensor = infer_graph.get_tensor_by_name('Identity:0')
-
-    data_sess = tf.compat.v1.Session(graph=data_graph,  config=data_config)
-    infer_sess = tf.compat.v1.Session(graph=infer_graph, config=infer_config)
-
-    num_processed_images = 0
-    num_remaining_images = IMAGENET_VALIDATION_IMAGES
-
-    if (not self.args.accuracy_only):
-      print("Benchmark")
-      iteration = 0
-      warm_up_iteration = self.args.warmup_steps
-      total_run = self.args.steps
-      total_time = 0
-
-      while num_remaining_images >= self.args.batch_size and iteration < total_run:
-        iteration += 1
-
-        # Reads and preprocess data
-        data_load_start = time.time()
-        image_np = data_sess.run(images)
-        data_load_time = time.time() - data_load_start
-
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        tf_logging.warn('---> Start iteration{0}'.format(str(iteration)))
-        start_time = time.time()
-        predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np})
-        time_consume = time.time() - start_time
-        tf_logging.warn('---> Stop iteration{0}'.format(str(iteration)))
-
-        # only add data loading time for real data, not for dummy data
-        if self.args.data_location:
-          time_consume += data_load_time
-
-        print('Iteration %d: %.6f sec' % (iteration, time_consume))
-        if iteration > warm_up_iteration:
-          total_time += time_consume
-
-      time_average = total_time / (iteration - warm_up_iteration)
-      print('Average time: %.6f sec' % (time_average))
-
-      print('Batch size = %d' % self.args.batch_size)
-      if (self.args.batch_size == 1):
-        print('Latency: %.3f ms' % (time_average * 1000))
-      # print throughput for both batch size 1 and 128
-      print('Throughput: %.3f images/sec' % (self.args.batch_size / time_average))
-
-    else: # accuracy check
-      print("Accuracy Check")
-      total_accuracy1, total_accuracy5 = (0.0, 0.0)
-
-      while num_remaining_images >= self.args.batch_size:
-        # Reads and preprocess data
-        np_images, np_labels = data_sess.run([images, labels])
-        np_labels -= 1
-        num_processed_images += self.args.batch_size
-        num_remaining_images -= self.args.batch_size
-
-        start_time = time.time()
-        # Compute inference on the preprocessed data
-        predictions = infer_sess.run(output_tensor,
-                               {input_tensor: np_images})
-        elapsed_time = time.time() - start_time
-        with tf.Graph().as_default() as accu_graph:
-          # Putting all code within this make things faster.
-          accuracy1 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=1), tf.float32))
-
-          accuracy5 = tf.reduce_sum(
-            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                 targets=tf.constant(np_labels), k=5), tf.float32))
-          with tf.compat.v1.Session() as accu_sess:
-            np_accuracy1, np_accuracy5 = accu_sess.run([accuracy1, accuracy5])
-          total_accuracy1 += np_accuracy1
-          total_accuracy5 += np_accuracy5
-          print("Iteration time: %0.4f ms" % elapsed_time)
-          print("Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
-              % (num_processed_images, total_accuracy1 / num_processed_images,
-                 total_accuracy5 / num_processed_images))
-
-  def validate_args(self):
-    """validate the arguments"""
-
-    if not self.args.data_location:
-      if self.args.accuracy_only:
-        raise ValueError("You must use real data for accuracy measurement.")
-
-
-if __name__ == "__main__":
-  evaluate_opt_graph = eval_classifier_optimized_graph()
-  evaluate_opt_graph.run()
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/datasets.py b/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/datasets.py
deleted file mode 100644
index d484936ee..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/datasets.py
+++ /dev/null
@@ -1,96 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Benchmark dataset utilities.
-"""
-
-from abc import abstractmethod
-import os
-
-import tensorflow as tf
-
-from inference.preprocess import preprocessing
-
-IMAGENET_NUM_TRAIN_IMAGES = 1281167
-IMAGENET_NUM_VAL_IMAGES = 50000
-IMAGENET_NUM_CLASSES = 1000
-
-class Dataset(object):
-  """Abstract class for cnn benchmarks dataset."""
-
-  def __init__(self, name, data_dir=None):
-    self.name = name
-    if data_dir is None:
-      raise ValueError('Data directory not specified')
-    self.data_dir = data_dir
-
-  def tf_record_pattern(self, subset):
-    return os.path.join(self.data_dir, '%s-*-of-*' % subset)
-
-  def reader(self):
-    return tf.compat.v1.TFRecordReader()
-
-  @abstractmethod
-  def num_classes(self):
-    pass
-
-  @abstractmethod
-  def num_examples_per_epoch(self, subset):
-    pass
-
-  def __str__(self):
-    return self.name
-
-
-class ImagenetData(Dataset):
-
-  def __init__(self, data_dir=None):
-    super(ImagenetData, self).__init__('ImageNet', data_dir)
-
-  def num_classes(self):
-    return IMAGENET_NUM_CLASSES
-
-  def num_examples_per_epoch(self, subset='train'):
-    if subset == 'train':
-      return IMAGENET_NUM_TRAIN_IMAGES
-    elif subset == 'validation':
-      return IMAGENET_NUM_VAL_IMAGES
-    elif subset == 'calibrate' or subset == 'calibration':
-      return 100
-    else:
-      raise ValueError('Invalid data subset "%s"' % subset)
-
-  def get_image_preprocessor(self):
-    return preprocessing.RecordInputImagePreprocessor
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/preprocessing.py b/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/preprocessing.py
deleted file mode 100644
index cc7442aa9..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/inference/preprocess/preprocessing.py
+++ /dev/null
@@ -1,164 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2023 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import tensorflow as tf
-from tensorflow.python.data.experimental import parallel_interleave
-from tensorflow.python.data.experimental import map_and_batch
-from tensorflow.python.platform import gfile
-import numpy as np
-
-
-def parse_example_proto(example_serialized):
-  """Parses an Example proto containing a training example of an image.
-  """
-  # Dense features in Example proto.
-  feature_map = {
-    'image/encoded': tf.io.FixedLenFeature([], dtype=tf.string,
-                                        default_value=''),
-    'image/class/label': tf.io.FixedLenFeature([1], dtype=tf.int64,
-                                            default_value=-1),
-  }
-  sparse_float32 = tf.io.VarLenFeature(dtype=tf.float32)
-  # Sparse features in Example proto.
-  feature_map.update(
-    {k: sparse_float32 for k in ['image/object/bbox/xmin',
-                                 'image/object/bbox/ymin',
-                                 'image/object/bbox/xmax',
-                                 'image/object/bbox/ymax']})
-
-  features = tf.io.parse_single_example(serialized=example_serialized, features=feature_map)
-  label = tf.cast(features['image/class/label'], dtype=tf.int32)
-
-  return features['image/encoded'], label
-
-
-def eval_image(image, height, width, resize_method,
-               central_fraction=0.875, scope=None):
-
-  with tf.compat.v1.name_scope('eval_image'):
-    print("Eval image")
-    if resize_method == 'crop':
-      image_resized = tf.image.resize(image, (224, 224))
-      image_resized = tf.cast(image_resized, tf.float32)
-      image_resized = (image_resized - 127.5) / 127.5
-      print(image_resized)
-      return image_resized
-    else:  # bilinear
-      if image.dtype != tf.float32:
-        image = tf.image.convert_image_dtype(image, dtype=tf.float32)
-      # Crop the central region of the image with an area containing 87.5% of
-      # the original image.
-      if central_fraction:
-        image = tf.image.central_crop(image, central_fraction=central_fraction)
-
-      if height and width:
-        # Resize the image to the specified height and width.
-        image = tf.expand_dims(image, 0)
-        image = tf.image.resize(image, [height, width],
-                                         method=tf.image.ResizeMethod.BILINEAR)
-        image = tf.squeeze(image, [0])
-      image = tf.subtract(image, 0.5)
-      image = tf.multiply(image, 2.0)
-      return image
-
-
-class RecordInputImagePreprocessor(object):
-  """Preprocessor for images with RecordInput format."""
-
-  def __init__(self,
-               height,
-               width,
-               batch_size,
-               intra_threads,
-               resize_method="bilinear"):
-
-    self.height = height
-    self.width = width
-    self.batch_size = batch_size
-    self.intra_threads = intra_threads
-    self.resize_method = resize_method
-
-  def parse_and_preprocess(self, value):
-    # parse
-    image_buffer, label_index = parse_example_proto(value)
-    # preprocess
-    image = tf.image.decode_jpeg(
-      image_buffer, channels=3, fancy_upscaling=False, dct_method='INTEGER_FAST')
-    image = eval_image(image, self.height, self.width, self.resize_method)
-
-    return (image, label_index)
-
-  def minibatch(self, dataset, subset, cache_data=False):
-
-    with tf.compat.v1.name_scope('batch_processing'):
-
-      glob_pattern = dataset.tf_record_pattern(subset)
-      file_names = gfile.Glob(glob_pattern)
-      if not file_names:
-        raise ValueError('Found no files in --data_dir matching: {}'
-                         .format(glob_pattern))
-      ds = tf.data.TFRecordDataset.list_files(file_names)
-
-      ds = ds.apply(
-        parallel_interleave(
-          tf.data.TFRecordDataset, cycle_length=28, block_length=5,
-          sloppy=True,
-          buffer_output_elements=10000, prefetch_input_elements=10000))
-
-      if cache_data:
-        ds = ds.take(1).cache().repeat()
-
-      ds = ds.prefetch(buffer_size=10000)
-      #ds = ds.prefetch(buffer_size=self.batch_size)
-
-      ds = ds.apply(
-        map_and_batch(
-          map_func=self.parse_and_preprocess,
-          batch_size=self.batch_size,
-          num_parallel_batches=56,
-          num_parallel_calls=None))
-
-      ds = ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
-
-      ds_iterator = tf.compat.v1.data.make_one_shot_iterator(ds)
-      images, labels = ds_iterator.get_next()
-      # reshape
-      labels = tf.reshape(labels, [self.batch_size])
-
-      return images, labels
diff --git a/models/image_recognition/tensorflow/vision_transformer/training/config.json b/models/image_recognition/tensorflow/vision_transformer/training/config.json
deleted file mode 100644
index 21c61d3f8..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/training/config.json
+++ /dev/null
@@ -1,2024 +0,0 @@
-{
-  "_name_or_path": "google/vit-base-patch16-224",
-  "architectures": [
-    "ViTForImageClassification"
-  ],
-  "attention_probs_dropout_prob": 0.0,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.0,
-  "hidden_size": 768,
-  "id2label": {
-    "0": "tench, Tinca tinca",
-    "1": "goldfish, Carassius auratus",
-    "2": "great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias",
-    "3": "tiger shark, Galeocerdo cuvieri",
-    "4": "hammerhead, hammerhead shark",
-    "5": "electric ray, crampfish, numbfish, torpedo",
-    "6": "stingray",
-    "7": "cock",
-    "8": "hen",
-    "9": "ostrich, Struthio camelus",
-    "10": "brambling, Fringilla montifringilla",
-    "11": "goldfinch, Carduelis carduelis",
-    "12": "house finch, linnet, Carpodacus mexicanus",
-    "13": "junco, snowbird",
-    "14": "indigo bunting, indigo finch, indigo bird, Passerina cyanea",
-    "15": "robin, American robin, Turdus migratorius",
-    "16": "bulbul",
-    "17": "jay",
-    "18": "magpie",
-    "19": "chickadee",
-    "20": "water ouzel, dipper",
-    "21": "kite",
-    "22": "bald eagle, American eagle, Haliaeetus leucocephalus",
-    "23": "vulture",
-    "24": "great grey owl, great gray owl, Strix nebulosa",
-    "25": "European fire salamander, Salamandra salamandra",
-    "26": "common newt, Triturus vulgaris",
-    "27": "eft",
-    "28": "spotted salamander, Ambystoma maculatum",
-    "29": "axolotl, mud puppy, Ambystoma mexicanum",
-    "30": "bullfrog, Rana catesbeiana",
-    "31": "tree frog, tree-frog",
-    "32": "tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui",
-    "33": "loggerhead, loggerhead turtle, Caretta caretta",
-    "34": "leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea",
-    "35": "mud turtle",
-    "36": "terrapin",
-    "37": "box turtle, box tortoise",
-    "38": "banded gecko",
-    "39": "common iguana, iguana, Iguana iguana",
-    "40": "American chameleon, anole, Anolis carolinensis",
-    "41": "whiptail, whiptail lizard",
-    "42": "agama",
-    "43": "frilled lizard, Chlamydosaurus kingi",
-    "44": "alligator lizard",
-    "45": "Gila monster, Heloderma suspectum",
-    "46": "green lizard, Lacerta viridis",
-    "47": "African chameleon, Chamaeleo chamaeleon",
-    "48": "Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis",
-    "49": "African crocodile, Nile crocodile, Crocodylus niloticus",
-    "50": "American alligator, Alligator mississipiensis",
-    "51": "triceratops",
-    "52": "thunder snake, worm snake, Carphophis amoenus",
-    "53": "ringneck snake, ring-necked snake, ring snake",
-    "54": "hognose snake, puff adder, sand viper",
-    "55": "green snake, grass snake",
-    "56": "king snake, kingsnake",
-    "57": "garter snake, grass snake",
-    "58": "water snake",
-    "59": "vine snake",
-    "60": "night snake, Hypsiglena torquata",
-    "61": "boa constrictor, Constrictor constrictor",
-    "62": "rock python, rock snake, Python sebae",
-    "63": "Indian cobra, Naja naja",
-    "64": "green mamba",
-    "65": "sea snake",
-    "66": "horned viper, cerastes, sand viper, horned asp, Cerastes cornutus",
-    "67": "diamondback, diamondback rattlesnake, Crotalus adamanteus",
-    "68": "sidewinder, horned rattlesnake, Crotalus cerastes",
-    "69": "trilobite",
-    "70": "harvestman, daddy longlegs, Phalangium opilio",
-    "71": "scorpion",
-    "72": "black and gold garden spider, Argiope aurantia",
-    "73": "barn spider, Araneus cavaticus",
-    "74": "garden spider, Aranea diademata",
-    "75": "black widow, Latrodectus mactans",
-    "76": "tarantula",
-    "77": "wolf spider, hunting spider",
-    "78": "tick",
-    "79": "centipede",
-    "80": "black grouse",
-    "81": "ptarmigan",
-    "82": "ruffed grouse, partridge, Bonasa umbellus",
-    "83": "prairie chicken, prairie grouse, prairie fowl",
-    "84": "peacock",
-    "85": "quail",
-    "86": "partridge",
-    "87": "African grey, African gray, Psittacus erithacus",
-    "88": "macaw",
-    "89": "sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita",
-    "90": "lorikeet",
-    "91": "coucal",
-    "92": "bee eater",
-    "93": "hornbill",
-    "94": "hummingbird",
-    "95": "jacamar",
-    "96": "toucan",
-    "97": "drake",
-    "98": "red-breasted merganser, Mergus serrator",
-    "99": "goose",
-    "100": "black swan, Cygnus atratus",
-    "101": "tusker",
-    "102": "echidna, spiny anteater, anteater",
-    "103": "platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus",
-    "104": "wallaby, brush kangaroo",
-    "105": "koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus",
-    "106": "wombat",
-    "107": "jellyfish",
-    "108": "sea anemone, anemone",
-    "109": "brain coral",
-    "110": "flatworm, platyhelminth",
-    "111": "nematode, nematode worm, roundworm",
-    "112": "conch",
-    "113": "snail",
-    "114": "slug",
-    "115": "sea slug, nudibranch",
-    "116": "chiton, coat-of-mail shell, sea cradle, polyplacophore",
-    "117": "chambered nautilus, pearly nautilus, nautilus",
-    "118": "Dungeness crab, Cancer magister",
-    "119": "rock crab, Cancer irroratus",
-    "120": "fiddler crab",
-    "121": "king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica",
-    "122": "American lobster, Northern lobster, Maine lobster, Homarus americanus",
-    "123": "spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish",
-    "124": "crayfish, crawfish, crawdad, crawdaddy",
-    "125": "hermit crab",
-    "126": "isopod",
-    "127": "white stork, Ciconia ciconia",
-    "128": "black stork, Ciconia nigra",
-    "129": "spoonbill",
-    "130": "flamingo",
-    "131": "little blue heron, Egretta caerulea",
-    "132": "American egret, great white heron, Egretta albus",
-    "133": "bittern",
-    "134": "crane",
-    "135": "limpkin, Aramus pictus",
-    "136": "European gallinule, Porphyrio porphyrio",
-    "137": "American coot, marsh hen, mud hen, water hen, Fulica americana",
-    "138": "bustard",
-    "139": "ruddy turnstone, Arenaria interpres",
-    "140": "red-backed sandpiper, dunlin, Erolia alpina",
-    "141": "redshank, Tringa totanus",
-    "142": "dowitcher",
-    "143": "oystercatcher, oyster catcher",
-    "144": "pelican",
-    "145": "king penguin, Aptenodytes patagonica",
-    "146": "albatross, mollymawk",
-    "147": "grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus",
-    "148": "killer whale, killer, orca, grampus, sea wolf, Orcinus orca",
-    "149": "dugong, Dugong dugon",
-    "150": "sea lion",
-    "151": "Chihuahua",
-    "152": "Japanese spaniel",
-    "153": "Maltese dog, Maltese terrier, Maltese",
-    "154": "Pekinese, Pekingese, Peke",
-    "155": "Shih-Tzu",
-    "156": "Blenheim spaniel",
-    "157": "papillon",
-    "158": "toy terrier",
-    "159": "Rhodesian ridgeback",
-    "160": "Afghan hound, Afghan",
-    "161": "basset, basset hound",
-    "162": "beagle",
-    "163": "bloodhound, sleuthhound",
-    "164": "bluetick",
-    "165": "black-and-tan coonhound",
-    "166": "Walker hound, Walker foxhound",
-    "167": "English foxhound",
-    "168": "redbone",
-    "169": "borzoi, Russian wolfhound",
-    "170": "Irish wolfhound",
-    "171": "Italian greyhound",
-    "172": "whippet",
-    "173": "Ibizan hound, Ibizan Podenco",
-    "174": "Norwegian elkhound, elkhound",
-    "175": "otterhound, otter hound",
-    "176": "Saluki, gazelle hound",
-    "177": "Scottish deerhound, deerhound",
-    "178": "Weimaraner",
-    "179": "Staffordshire bullterrier, Staffordshire bull terrier",
-    "180": "American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier",
-    "181": "Bedlington terrier",
-    "182": "Border terrier",
-    "183": "Kerry blue terrier",
-    "184": "Irish terrier",
-    "185": "Norfolk terrier",
-    "186": "Norwich terrier",
-    "187": "Yorkshire terrier",
-    "188": "wire-haired fox terrier",
-    "189": "Lakeland terrier",
-    "190": "Sealyham terrier, Sealyham",
-    "191": "Airedale, Airedale terrier",
-    "192": "cairn, cairn terrier",
-    "193": "Australian terrier",
-    "194": "Dandie Dinmont, Dandie Dinmont terrier",
-    "195": "Boston bull, Boston terrier",
-    "196": "miniature schnauzer",
-    "197": "giant schnauzer",
-    "198": "standard schnauzer",
-    "199": "Scotch terrier, Scottish terrier, Scottie",
-    "200": "Tibetan terrier, chrysanthemum dog",
-    "201": "silky terrier, Sydney silky",
-    "202": "soft-coated wheaten terrier",
-    "203": "West Highland white terrier",
-    "204": "Lhasa, Lhasa apso",
-    "205": "flat-coated retriever",
-    "206": "curly-coated retriever",
-    "207": "golden retriever",
-    "208": "Labrador retriever",
-    "209": "Chesapeake Bay retriever",
-    "210": "German short-haired pointer",
-    "211": "vizsla, Hungarian pointer",
-    "212": "English setter",
-    "213": "Irish setter, red setter",
-    "214": "Gordon setter",
-    "215": "Brittany spaniel",
-    "216": "clumber, clumber spaniel",
-    "217": "English springer, English springer spaniel",
-    "218": "Welsh springer spaniel",
-    "219": "cocker spaniel, English cocker spaniel, cocker",
-    "220": "Sussex spaniel",
-    "221": "Irish water spaniel",
-    "222": "kuvasz",
-    "223": "schipperke",
-    "224": "groenendael",
-    "225": "malinois",
-    "226": "briard",
-    "227": "kelpie",
-    "228": "komondor",
-    "229": "Old English sheepdog, bobtail",
-    "230": "Shetland sheepdog, Shetland sheep dog, Shetland",
-    "231": "collie",
-    "232": "Border collie",
-    "233": "Bouvier des Flandres, Bouviers des Flandres",
-    "234": "Rottweiler",
-    "235": "German shepherd, German shepherd dog, German police dog, alsatian",
-    "236": "Doberman, Doberman pinscher",
-    "237": "miniature pinscher",
-    "238": "Greater Swiss Mountain dog",
-    "239": "Bernese mountain dog",
-    "240": "Appenzeller",
-    "241": "EntleBucher",
-    "242": "boxer",
-    "243": "bull mastiff",
-    "244": "Tibetan mastiff",
-    "245": "French bulldog",
-    "246": "Great Dane",
-    "247": "Saint Bernard, St Bernard",
-    "248": "Eskimo dog, husky",
-    "249": "malamute, malemute, Alaskan malamute",
-    "250": "Siberian husky",
-    "251": "dalmatian, coach dog, carriage dog",
-    "252": "affenpinscher, monkey pinscher, monkey dog",
-    "253": "basenji",
-    "254": "pug, pug-dog",
-    "255": "Leonberg",
-    "256": "Newfoundland, Newfoundland dog",
-    "257": "Great Pyrenees",
-    "258": "Samoyed, Samoyede",
-    "259": "Pomeranian",
-    "260": "chow, chow chow",
-    "261": "keeshond",
-    "262": "Brabancon griffon",
-    "263": "Pembroke, Pembroke Welsh corgi",
-    "264": "Cardigan, Cardigan Welsh corgi",
-    "265": "toy poodle",
-    "266": "miniature poodle",
-    "267": "standard poodle",
-    "268": "Mexican hairless",
-    "269": "timber wolf, grey wolf, gray wolf, Canis lupus",
-    "270": "white wolf, Arctic wolf, Canis lupus tundrarum",
-    "271": "red wolf, maned wolf, Canis rufus, Canis niger",
-    "272": "coyote, prairie wolf, brush wolf, Canis latrans",
-    "273": "dingo, warrigal, warragal, Canis dingo",
-    "274": "dhole, Cuon alpinus",
-    "275": "African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus",
-    "276": "hyena, hyaena",
-    "277": "red fox, Vulpes vulpes",
-    "278": "kit fox, Vulpes macrotis",
-    "279": "Arctic fox, white fox, Alopex lagopus",
-    "280": "grey fox, gray fox, Urocyon cinereoargenteus",
-    "281": "tabby, tabby cat",
-    "282": "tiger cat",
-    "283": "Persian cat",
-    "284": "Siamese cat, Siamese",
-    "285": "Egyptian cat",
-    "286": "cougar, puma, catamount, mountain lion, painter, panther, Felis concolor",
-    "287": "lynx, catamount",
-    "288": "leopard, Panthera pardus",
-    "289": "snow leopard, ounce, Panthera uncia",
-    "290": "jaguar, panther, Panthera onca, Felis onca",
-    "291": "lion, king of beasts, Panthera leo",
-    "292": "tiger, Panthera tigris",
-    "293": "cheetah, chetah, Acinonyx jubatus",
-    "294": "brown bear, bruin, Ursus arctos",
-    "295": "American black bear, black bear, Ursus americanus, Euarctos americanus",
-    "296": "ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus",
-    "297": "sloth bear, Melursus ursinus, Ursus ursinus",
-    "298": "mongoose",
-    "299": "meerkat, mierkat",
-    "300": "tiger beetle",
-    "301": "ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle",
-    "302": "ground beetle, carabid beetle",
-    "303": "long-horned beetle, longicorn, longicorn beetle",
-    "304": "leaf beetle, chrysomelid",
-    "305": "dung beetle",
-    "306": "rhinoceros beetle",
-    "307": "weevil",
-    "308": "fly",
-    "309": "bee",
-    "310": "ant, emmet, pismire",
-    "311": "grasshopper, hopper",
-    "312": "cricket",
-    "313": "walking stick, walkingstick, stick insect",
-    "314": "cockroach, roach",
-    "315": "mantis, mantid",
-    "316": "cicada, cicala",
-    "317": "leafhopper",
-    "318": "lacewing, lacewing fly",
-    "319": "dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk",
-    "320": "damselfly",
-    "321": "admiral",
-    "322": "ringlet, ringlet butterfly",
-    "323": "monarch, monarch butterfly, milkweed butterfly, Danaus plexippus",
-    "324": "cabbage butterfly",
-    "325": "sulphur butterfly, sulfur butterfly",
-    "326": "lycaenid, lycaenid butterfly",
-    "327": "starfish, sea star",
-    "328": "sea urchin",
-    "329": "sea cucumber, holothurian",
-    "330": "wood rabbit, cottontail, cottontail rabbit",
-    "331": "hare",
-    "332": "Angora, Angora rabbit",
-    "333": "hamster",
-    "334": "porcupine, hedgehog",
-    "335": "fox squirrel, eastern fox squirrel, Sciurus niger",
-    "336": "marmot",
-    "337": "beaver",
-    "338": "guinea pig, Cavia cobaya",
-    "339": "sorrel",
-    "340": "zebra",
-    "341": "hog, pig, grunter, squealer, Sus scrofa",
-    "342": "wild boar, boar, Sus scrofa",
-    "343": "warthog",
-    "344": "hippopotamus, hippo, river horse, Hippopotamus amphibius",
-    "345": "ox",
-    "346": "water buffalo, water ox, Asiatic buffalo, Bubalus bubalis",
-    "347": "bison",
-    "348": "ram, tup",
-    "349": "bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis",
-    "350": "ibex, Capra ibex",
-    "351": "hartebeest",
-    "352": "impala, Aepyceros melampus",
-    "353": "gazelle",
-    "354": "Arabian camel, dromedary, Camelus dromedarius",
-    "355": "llama",
-    "356": "weasel",
-    "357": "mink",
-    "358": "polecat, fitch, foulmart, foumart, Mustela putorius",
-    "359": "black-footed ferret, ferret, Mustela nigripes",
-    "360": "otter",
-    "361": "skunk, polecat, wood pussy",
-    "362": "badger",
-    "363": "armadillo",
-    "364": "three-toed sloth, ai, Bradypus tridactylus",
-    "365": "orangutan, orang, orangutang, Pongo pygmaeus",
-    "366": "gorilla, Gorilla gorilla",
-    "367": "chimpanzee, chimp, Pan troglodytes",
-    "368": "gibbon, Hylobates lar",
-    "369": "siamang, Hylobates syndactylus, Symphalangus syndactylus",
-    "370": "guenon, guenon monkey",
-    "371": "patas, hussar monkey, Erythrocebus patas",
-    "372": "baboon",
-    "373": "macaque",
-    "374": "langur",
-    "375": "colobus, colobus monkey",
-    "376": "proboscis monkey, Nasalis larvatus",
-    "377": "marmoset",
-    "378": "capuchin, ringtail, Cebus capucinus",
-    "379": "howler monkey, howler",
-    "380": "titi, titi monkey",
-    "381": "spider monkey, Ateles geoffroyi",
-    "382": "squirrel monkey, Saimiri sciureus",
-    "383": "Madagascar cat, ring-tailed lemur, Lemur catta",
-    "384": "indri, indris, Indri indri, Indri brevicaudatus",
-    "385": "Indian elephant, Elephas maximus",
-    "386": "African elephant, Loxodonta africana",
-    "387": "lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens",
-    "388": "giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca",
-    "389": "barracouta, snoek",
-    "390": "eel",
-    "391": "coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch",
-    "392": "rock beauty, Holocanthus tricolor",
-    "393": "anemone fish",
-    "394": "sturgeon",
-    "395": "gar, garfish, garpike, billfish, Lepisosteus osseus",
-    "396": "lionfish",
-    "397": "puffer, pufferfish, blowfish, globefish",
-    "398": "abacus",
-    "399": "abaya",
-    "400": "academic gown, academic robe, judge's robe",
-    "401": "accordion, piano accordion, squeeze box",
-    "402": "acoustic guitar",
-    "403": "aircraft carrier, carrier, flattop, attack aircraft carrier",
-    "404": "airliner",
-    "405": "airship, dirigible",
-    "406": "altar",
-    "407": "ambulance",
-    "408": "amphibian, amphibious vehicle",
-    "409": "analog clock",
-    "410": "apiary, bee house",
-    "411": "apron",
-    "412": "ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin",
-    "413": "assault rifle, assault gun",
-    "414": "backpack, back pack, knapsack, packsack, rucksack, haversack",
-    "415": "bakery, bakeshop, bakehouse",
-    "416": "balance beam, beam",
-    "417": "balloon",
-    "418": "ballpoint, ballpoint pen, ballpen, Biro",
-    "419": "Band Aid",
-    "420": "banjo",
-    "421": "bannister, banister, balustrade, balusters, handrail",
-    "422": "barbell",
-    "423": "barber chair",
-    "424": "barbershop",
-    "425": "barn",
-    "426": "barometer",
-    "427": "barrel, cask",
-    "428": "barrow, garden cart, lawn cart, wheelbarrow",
-    "429": "baseball",
-    "430": "basketball",
-    "431": "bassinet",
-    "432": "bassoon",
-    "433": "bathing cap, swimming cap",
-    "434": "bath towel",
-    "435": "bathtub, bathing tub, bath, tub",
-    "436": "beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon",
-    "437": "beacon, lighthouse, beacon light, pharos",
-    "438": "beaker",
-    "439": "bearskin, busby, shako",
-    "440": "beer bottle",
-    "441": "beer glass",
-    "442": "bell cote, bell cot",
-    "443": "bib",
-    "444": "bicycle-built-for-two, tandem bicycle, tandem",
-    "445": "bikini, two-piece",
-    "446": "binder, ring-binder",
-    "447": "binoculars, field glasses, opera glasses",
-    "448": "birdhouse",
-    "449": "boathouse",
-    "450": "bobsled, bobsleigh, bob",
-    "451": "bolo tie, bolo, bola tie, bola",
-    "452": "bonnet, poke bonnet",
-    "453": "bookcase",
-    "454": "bookshop, bookstore, bookstall",
-    "455": "bottlecap",
-    "456": "bow",
-    "457": "bow tie, bow-tie, bowtie",
-    "458": "brass, memorial tablet, plaque",
-    "459": "brassiere, bra, bandeau",
-    "460": "breakwater, groin, groyne, mole, bulwark, seawall, jetty",
-    "461": "breastplate, aegis, egis",
-    "462": "broom",
-    "463": "bucket, pail",
-    "464": "buckle",
-    "465": "bulletproof vest",
-    "466": "bullet train, bullet",
-    "467": "butcher shop, meat market",
-    "468": "cab, hack, taxi, taxicab",
-    "469": "caldron, cauldron",
-    "470": "candle, taper, wax light",
-    "471": "cannon",
-    "472": "canoe",
-    "473": "can opener, tin opener",
-    "474": "cardigan",
-    "475": "car mirror",
-    "476": "carousel, carrousel, merry-go-round, roundabout, whirligig",
-    "477": "carpenter's kit, tool kit",
-    "478": "carton",
-    "479": "car wheel",
-    "480": "cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM",
-    "481": "cassette",
-    "482": "cassette player",
-    "483": "castle",
-    "484": "catamaran",
-    "485": "CD player",
-    "486": "cello, violoncello",
-    "487": "cellular telephone, cellular phone, cellphone, cell, mobile phone",
-    "488": "chain",
-    "489": "chainlink fence",
-    "490": "chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour",
-    "491": "chain saw, chainsaw",
-    "492": "chest",
-    "493": "chiffonier, commode",
-    "494": "chime, bell, gong",
-    "495": "china cabinet, china closet",
-    "496": "Christmas stocking",
-    "497": "church, church building",
-    "498": "cinema, movie theater, movie theatre, movie house, picture palace",
-    "499": "cleaver, meat cleaver, chopper",
-    "500": "cliff dwelling",
-    "501": "cloak",
-    "502": "clog, geta, patten, sabot",
-    "503": "cocktail shaker",
-    "504": "coffee mug",
-    "505": "coffeepot",
-    "506": "coil, spiral, volute, whorl, helix",
-    "507": "combination lock",
-    "508": "computer keyboard, keypad",
-    "509": "confectionery, confectionary, candy store",
-    "510": "container ship, containership, container vessel",
-    "511": "convertible",
-    "512": "corkscrew, bottle screw",
-    "513": "cornet, horn, trumpet, trump",
-    "514": "cowboy boot",
-    "515": "cowboy hat, ten-gallon hat",
-    "516": "cradle",
-    "517": "crane",
-    "518": "crash helmet",
-    "519": "crate",
-    "520": "crib, cot",
-    "521": "Crock Pot",
-    "522": "croquet ball",
-    "523": "crutch",
-    "524": "cuirass",
-    "525": "dam, dike, dyke",
-    "526": "desk",
-    "527": "desktop computer",
-    "528": "dial telephone, dial phone",
-    "529": "diaper, nappy, napkin",
-    "530": "digital clock",
-    "531": "digital watch",
-    "532": "dining table, board",
-    "533": "dishrag, dishcloth",
-    "534": "dishwasher, dish washer, dishwashing machine",
-    "535": "disk brake, disc brake",
-    "536": "dock, dockage, docking facility",
-    "537": "dogsled, dog sled, dog sleigh",
-    "538": "dome",
-    "539": "doormat, welcome mat",
-    "540": "drilling platform, offshore rig",
-    "541": "drum, membranophone, tympan",
-    "542": "drumstick",
-    "543": "dumbbell",
-    "544": "Dutch oven",
-    "545": "electric fan, blower",
-    "546": "electric guitar",
-    "547": "electric locomotive",
-    "548": "entertainment center",
-    "549": "envelope",
-    "550": "espresso maker",
-    "551": "face powder",
-    "552": "feather boa, boa",
-    "553": "file, file cabinet, filing cabinet",
-    "554": "fireboat",
-    "555": "fire engine, fire truck",
-    "556": "fire screen, fireguard",
-    "557": "flagpole, flagstaff",
-    "558": "flute, transverse flute",
-    "559": "folding chair",
-    "560": "football helmet",
-    "561": "forklift",
-    "562": "fountain",
-    "563": "fountain pen",
-    "564": "four-poster",
-    "565": "freight car",
-    "566": "French horn, horn",
-    "567": "frying pan, frypan, skillet",
-    "568": "fur coat",
-    "569": "garbage truck, dustcart",
-    "570": "gasmask, respirator, gas helmet",
-    "571": "gas pump, gasoline pump, petrol pump, island dispenser",
-    "572": "goblet",
-    "573": "go-kart",
-    "574": "golf ball",
-    "575": "golfcart, golf cart",
-    "576": "gondola",
-    "577": "gong, tam-tam",
-    "578": "gown",
-    "579": "grand piano, grand",
-    "580": "greenhouse, nursery, glasshouse",
-    "581": "grille, radiator grille",
-    "582": "grocery store, grocery, food market, market",
-    "583": "guillotine",
-    "584": "hair slide",
-    "585": "hair spray",
-    "586": "half track",
-    "587": "hammer",
-    "588": "hamper",
-    "589": "hand blower, blow dryer, blow drier, hair dryer, hair drier",
-    "590": "hand-held computer, hand-held microcomputer",
-    "591": "handkerchief, hankie, hanky, hankey",
-    "592": "hard disc, hard disk, fixed disk",
-    "593": "harmonica, mouth organ, harp, mouth harp",
-    "594": "harp",
-    "595": "harvester, reaper",
-    "596": "hatchet",
-    "597": "holster",
-    "598": "home theater, home theatre",
-    "599": "honeycomb",
-    "600": "hook, claw",
-    "601": "hoopskirt, crinoline",
-    "602": "horizontal bar, high bar",
-    "603": "horse cart, horse-cart",
-    "604": "hourglass",
-    "605": "iPod",
-    "606": "iron, smoothing iron",
-    "607": "jack-o'-lantern",
-    "608": "jean, blue jean, denim",
-    "609": "jeep, landrover",
-    "610": "jersey, T-shirt, tee shirt",
-    "611": "jigsaw puzzle",
-    "612": "jinrikisha, ricksha, rickshaw",
-    "613": "joystick",
-    "614": "kimono",
-    "615": "knee pad",
-    "616": "knot",
-    "617": "lab coat, laboratory coat",
-    "618": "ladle",
-    "619": "lampshade, lamp shade",
-    "620": "laptop, laptop computer",
-    "621": "lawn mower, mower",
-    "622": "lens cap, lens cover",
-    "623": "letter opener, paper knife, paperknife",
-    "624": "library",
-    "625": "lifeboat",
-    "626": "lighter, light, igniter, ignitor",
-    "627": "limousine, limo",
-    "628": "liner, ocean liner",
-    "629": "lipstick, lip rouge",
-    "630": "Loafer",
-    "631": "lotion",
-    "632": "loudspeaker, speaker, speaker unit, loudspeaker system, speaker system",
-    "633": "loupe, jeweler's loupe",
-    "634": "lumbermill, sawmill",
-    "635": "magnetic compass",
-    "636": "mailbag, postbag",
-    "637": "mailbox, letter box",
-    "638": "maillot",
-    "639": "maillot, tank suit",
-    "640": "manhole cover",
-    "641": "maraca",
-    "642": "marimba, xylophone",
-    "643": "mask",
-    "644": "matchstick",
-    "645": "maypole",
-    "646": "maze, labyrinth",
-    "647": "measuring cup",
-    "648": "medicine chest, medicine cabinet",
-    "649": "megalith, megalithic structure",
-    "650": "microphone, mike",
-    "651": "microwave, microwave oven",
-    "652": "military uniform",
-    "653": "milk can",
-    "654": "minibus",
-    "655": "miniskirt, mini",
-    "656": "minivan",
-    "657": "missile",
-    "658": "mitten",
-    "659": "mixing bowl",
-    "660": "mobile home, manufactured home",
-    "661": "Model T",
-    "662": "modem",
-    "663": "monastery",
-    "664": "monitor",
-    "665": "moped",
-    "666": "mortar",
-    "667": "mortarboard",
-    "668": "mosque",
-    "669": "mosquito net",
-    "670": "motor scooter, scooter",
-    "671": "mountain bike, all-terrain bike, off-roader",
-    "672": "mountain tent",
-    "673": "mouse, computer mouse",
-    "674": "mousetrap",
-    "675": "moving van",
-    "676": "muzzle",
-    "677": "nail",
-    "678": "neck brace",
-    "679": "necklace",
-    "680": "nipple",
-    "681": "notebook, notebook computer",
-    "682": "obelisk",
-    "683": "oboe, hautboy, hautbois",
-    "684": "ocarina, sweet potato",
-    "685": "odometer, hodometer, mileometer, milometer",
-    "686": "oil filter",
-    "687": "organ, pipe organ",
-    "688": "oscilloscope, scope, cathode-ray oscilloscope, CRO",
-    "689": "overskirt",
-    "690": "oxcart",
-    "691": "oxygen mask",
-    "692": "packet",
-    "693": "paddle, boat paddle",
-    "694": "paddlewheel, paddle wheel",
-    "695": "padlock",
-    "696": "paintbrush",
-    "697": "pajama, pyjama, pj's, jammies",
-    "698": "palace",
-    "699": "panpipe, pandean pipe, syrinx",
-    "700": "paper towel",
-    "701": "parachute, chute",
-    "702": "parallel bars, bars",
-    "703": "park bench",
-    "704": "parking meter",
-    "705": "passenger car, coach, carriage",
-    "706": "patio, terrace",
-    "707": "pay-phone, pay-station",
-    "708": "pedestal, plinth, footstall",
-    "709": "pencil box, pencil case",
-    "710": "pencil sharpener",
-    "711": "perfume, essence",
-    "712": "Petri dish",
-    "713": "photocopier",
-    "714": "pick, plectrum, plectron",
-    "715": "pickelhaube",
-    "716": "picket fence, paling",
-    "717": "pickup, pickup truck",
-    "718": "pier",
-    "719": "piggy bank, penny bank",
-    "720": "pill bottle",
-    "721": "pillow",
-    "722": "ping-pong ball",
-    "723": "pinwheel",
-    "724": "pirate, pirate ship",
-    "725": "pitcher, ewer",
-    "726": "plane, carpenter's plane, woodworking plane",
-    "727": "planetarium",
-    "728": "plastic bag",
-    "729": "plate rack",
-    "730": "plow, plough",
-    "731": "plunger, plumber's helper",
-    "732": "Polaroid camera, Polaroid Land camera",
-    "733": "pole",
-    "734": "police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria",
-    "735": "poncho",
-    "736": "pool table, billiard table, snooker table",
-    "737": "pop bottle, soda bottle",
-    "738": "pot, flowerpot",
-    "739": "potter's wheel",
-    "740": "power drill",
-    "741": "prayer rug, prayer mat",
-    "742": "printer",
-    "743": "prison, prison house",
-    "744": "projectile, missile",
-    "745": "projector",
-    "746": "puck, hockey puck",
-    "747": "punching bag, punch bag, punching ball, punchball",
-    "748": "purse",
-    "749": "quill, quill pen",
-    "750": "quilt, comforter, comfort, puff",
-    "751": "racer, race car, racing car",
-    "752": "racket, racquet",
-    "753": "radiator",
-    "754": "radio, wireless",
-    "755": "radio telescope, radio reflector",
-    "756": "rain barrel",
-    "757": "recreational vehicle, RV, R.V.",
-    "758": "reel",
-    "759": "reflex camera",
-    "760": "refrigerator, icebox",
-    "761": "remote control, remote",
-    "762": "restaurant, eating house, eating place, eatery",
-    "763": "revolver, six-gun, six-shooter",
-    "764": "rifle",
-    "765": "rocking chair, rocker",
-    "766": "rotisserie",
-    "767": "rubber eraser, rubber, pencil eraser",
-    "768": "rugby ball",
-    "769": "rule, ruler",
-    "770": "running shoe",
-    "771": "safe",
-    "772": "safety pin",
-    "773": "saltshaker, salt shaker",
-    "774": "sandal",
-    "775": "sarong",
-    "776": "sax, saxophone",
-    "777": "scabbard",
-    "778": "scale, weighing machine",
-    "779": "school bus",
-    "780": "schooner",
-    "781": "scoreboard",
-    "782": "screen, CRT screen",
-    "783": "screw",
-    "784": "screwdriver",
-    "785": "seat belt, seatbelt",
-    "786": "sewing machine",
-    "787": "shield, buckler",
-    "788": "shoe shop, shoe-shop, shoe store",
-    "789": "shoji",
-    "790": "shopping basket",
-    "791": "shopping cart",
-    "792": "shovel",
-    "793": "shower cap",
-    "794": "shower curtain",
-    "795": "ski",
-    "796": "ski mask",
-    "797": "sleeping bag",
-    "798": "slide rule, slipstick",
-    "799": "sliding door",
-    "800": "slot, one-armed bandit",
-    "801": "snorkel",
-    "802": "snowmobile",
-    "803": "snowplow, snowplough",
-    "804": "soap dispenser",
-    "805": "soccer ball",
-    "806": "sock",
-    "807": "solar dish, solar collector, solar furnace",
-    "808": "sombrero",
-    "809": "soup bowl",
-    "810": "space bar",
-    "811": "space heater",
-    "812": "space shuttle",
-    "813": "spatula",
-    "814": "speedboat",
-    "815": "spider web, spider's web",
-    "816": "spindle",
-    "817": "sports car, sport car",
-    "818": "spotlight, spot",
-    "819": "stage",
-    "820": "steam locomotive",
-    "821": "steel arch bridge",
-    "822": "steel drum",
-    "823": "stethoscope",
-    "824": "stole",
-    "825": "stone wall",
-    "826": "stopwatch, stop watch",
-    "827": "stove",
-    "828": "strainer",
-    "829": "streetcar, tram, tramcar, trolley, trolley car",
-    "830": "stretcher",
-    "831": "studio couch, day bed",
-    "832": "stupa, tope",
-    "833": "submarine, pigboat, sub, U-boat",
-    "834": "suit, suit of clothes",
-    "835": "sundial",
-    "836": "sunglass",
-    "837": "sunglasses, dark glasses, shades",
-    "838": "sunscreen, sunblock, sun blocker",
-    "839": "suspension bridge",
-    "840": "swab, swob, mop",
-    "841": "sweatshirt",
-    "842": "swimming trunks, bathing trunks",
-    "843": "swing",
-    "844": "switch, electric switch, electrical switch",
-    "845": "syringe",
-    "846": "table lamp",
-    "847": "tank, army tank, armored combat vehicle, armoured combat vehicle",
-    "848": "tape player",
-    "849": "teapot",
-    "850": "teddy, teddy bear",
-    "851": "television, television system",
-    "852": "tennis ball",
-    "853": "thatch, thatched roof",
-    "854": "theater curtain, theatre curtain",
-    "855": "thimble",
-    "856": "thresher, thrasher, threshing machine",
-    "857": "throne",
-    "858": "tile roof",
-    "859": "toaster",
-    "860": "tobacco shop, tobacconist shop, tobacconist",
-    "861": "toilet seat",
-    "862": "torch",
-    "863": "totem pole",
-    "864": "tow truck, tow car, wrecker",
-    "865": "toyshop",
-    "866": "tractor",
-    "867": "trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi",
-    "868": "tray",
-    "869": "trench coat",
-    "870": "tricycle, trike, velocipede",
-    "871": "trimaran",
-    "872": "tripod",
-    "873": "triumphal arch",
-    "874": "trolleybus, trolley coach, trackless trolley",
-    "875": "trombone",
-    "876": "tub, vat",
-    "877": "turnstile",
-    "878": "typewriter keyboard",
-    "879": "umbrella",
-    "880": "unicycle, monocycle",
-    "881": "upright, upright piano",
-    "882": "vacuum, vacuum cleaner",
-    "883": "vase",
-    "884": "vault",
-    "885": "velvet",
-    "886": "vending machine",
-    "887": "vestment",
-    "888": "viaduct",
-    "889": "violin, fiddle",
-    "890": "volleyball",
-    "891": "waffle iron",
-    "892": "wall clock",
-    "893": "wallet, billfold, notecase, pocketbook",
-    "894": "wardrobe, closet, press",
-    "895": "warplane, military plane",
-    "896": "washbasin, handbasin, washbowl, lavabo, wash-hand basin",
-    "897": "washer, automatic washer, washing machine",
-    "898": "water bottle",
-    "899": "water jug",
-    "900": "water tower",
-    "901": "whiskey jug",
-    "902": "whistle",
-    "903": "wig",
-    "904": "window screen",
-    "905": "window shade",
-    "906": "Windsor tie",
-    "907": "wine bottle",
-    "908": "wing",
-    "909": "wok",
-    "910": "wooden spoon",
-    "911": "wool, woolen, woollen",
-    "912": "worm fence, snake fence, snake-rail fence, Virginia fence",
-    "913": "wreck",
-    "914": "yawl",
-    "915": "yurt",
-    "916": "web site, website, internet site, site",
-    "917": "comic book",
-    "918": "crossword puzzle, crossword",
-    "919": "street sign",
-    "920": "traffic light, traffic signal, stoplight",
-    "921": "book jacket, dust cover, dust jacket, dust wrapper",
-    "922": "menu",
-    "923": "plate",
-    "924": "guacamole",
-    "925": "consomme",
-    "926": "hot pot, hotpot",
-    "927": "trifle",
-    "928": "ice cream, icecream",
-    "929": "ice lolly, lolly, lollipop, popsicle",
-    "930": "French loaf",
-    "931": "bagel, beigel",
-    "932": "pretzel",
-    "933": "cheeseburger",
-    "934": "hotdog, hot dog, red hot",
-    "935": "mashed potato",
-    "936": "head cabbage",
-    "937": "broccoli",
-    "938": "cauliflower",
-    "939": "zucchini, courgette",
-    "940": "spaghetti squash",
-    "941": "acorn squash",
-    "942": "butternut squash",
-    "943": "cucumber, cuke",
-    "944": "artichoke, globe artichoke",
-    "945": "bell pepper",
-    "946": "cardoon",
-    "947": "mushroom",
-    "948": "Granny Smith",
-    "949": "strawberry",
-    "950": "orange",
-    "951": "lemon",
-    "952": "fig",
-    "953": "pineapple, ananas",
-    "954": "banana",
-    "955": "jackfruit, jak, jack",
-    "956": "custard apple",
-    "957": "pomegranate",
-    "958": "hay",
-    "959": "carbonara",
-    "960": "chocolate sauce, chocolate syrup",
-    "961": "dough",
-    "962": "meat loaf, meatloaf",
-    "963": "pizza, pizza pie",
-    "964": "potpie",
-    "965": "burrito",
-    "966": "red wine",
-    "967": "espresso",
-    "968": "cup",
-    "969": "eggnog",
-    "970": "alp",
-    "971": "bubble",
-    "972": "cliff, drop, drop-off",
-    "973": "coral reef",
-    "974": "geyser",
-    "975": "lakeside, lakeshore",
-    "976": "promontory, headland, head, foreland",
-    "977": "sandbar, sand bar",
-    "978": "seashore, coast, seacoast, sea-coast",
-    "979": "valley, vale",
-    "980": "volcano",
-    "981": "ballplayer, baseball player",
-    "982": "groom, bridegroom",
-    "983": "scuba diver",
-    "984": "rapeseed",
-    "985": "daisy",
-    "986": "yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum",
-    "987": "corn",
-    "988": "acorn",
-    "989": "hip, rose hip, rosehip",
-    "990": "buckeye, horse chestnut, conker",
-    "991": "coral fungus",
-    "992": "agaric",
-    "993": "gyromitra",
-    "994": "stinkhorn, carrion fungus",
-    "995": "earthstar",
-    "996": "hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa",
-    "997": "bolete",
-    "998": "ear, spike, capitulum",
-    "999": "toilet tissue, toilet paper, bathroom tissue"
-  },
-  "image_size": 224,
-  "initializer_range": 0.02,
-  "intermediate_size": 3072,
-  "label2id": {
-    "Afghan hound, Afghan": 160,
-    "African chameleon, Chamaeleo chamaeleon": 47,
-    "African crocodile, Nile crocodile, Crocodylus niloticus": 49,
-    "African elephant, Loxodonta africana": 386,
-    "African grey, African gray, Psittacus erithacus": 87,
-    "African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus": 275,
-    "Airedale, Airedale terrier": 191,
-    "American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier": 180,
-    "American alligator, Alligator mississipiensis": 50,
-    "American black bear, black bear, Ursus americanus, Euarctos americanus": 295,
-    "American chameleon, anole, Anolis carolinensis": 40,
-    "American coot, marsh hen, mud hen, water hen, Fulica americana": 137,
-    "American egret, great white heron, Egretta albus": 132,
-    "American lobster, Northern lobster, Maine lobster, Homarus americanus": 122,
-    "Angora, Angora rabbit": 332,
-    "Appenzeller": 240,
-    "Arabian camel, dromedary, Camelus dromedarius": 354,
-    "Arctic fox, white fox, Alopex lagopus": 279,
-    "Australian terrier": 193,
-    "Band Aid": 419,
-    "Bedlington terrier": 181,
-    "Bernese mountain dog": 239,
-    "Blenheim spaniel": 156,
-    "Border collie": 232,
-    "Border terrier": 182,
-    "Boston bull, Boston terrier": 195,
-    "Bouvier des Flandres, Bouviers des Flandres": 233,
-    "Brabancon griffon": 262,
-    "Brittany spaniel": 215,
-    "CD player": 485,
-    "Cardigan, Cardigan Welsh corgi": 264,
-    "Chesapeake Bay retriever": 209,
-    "Chihuahua": 151,
-    "Christmas stocking": 496,
-    "Crock Pot": 521,
-    "Dandie Dinmont, Dandie Dinmont terrier": 194,
-    "Doberman, Doberman pinscher": 236,
-    "Dungeness crab, Cancer magister": 118,
-    "Dutch oven": 544,
-    "Egyptian cat": 285,
-    "English foxhound": 167,
-    "English setter": 212,
-    "English springer, English springer spaniel": 217,
-    "EntleBucher": 241,
-    "Eskimo dog, husky": 248,
-    "European fire salamander, Salamandra salamandra": 25,
-    "European gallinule, Porphyrio porphyrio": 136,
-    "French bulldog": 245,
-    "French horn, horn": 566,
-    "French loaf": 930,
-    "German shepherd, German shepherd dog, German police dog, alsatian": 235,
-    "German short-haired pointer": 210,
-    "Gila monster, Heloderma suspectum": 45,
-    "Gordon setter": 214,
-    "Granny Smith": 948,
-    "Great Dane": 246,
-    "Great Pyrenees": 257,
-    "Greater Swiss Mountain dog": 238,
-    "Ibizan hound, Ibizan Podenco": 173,
-    "Indian cobra, Naja naja": 63,
-    "Indian elephant, Elephas maximus": 385,
-    "Irish setter, red setter": 213,
-    "Irish terrier": 184,
-    "Irish water spaniel": 221,
-    "Irish wolfhound": 170,
-    "Italian greyhound": 171,
-    "Japanese spaniel": 152,
-    "Kerry blue terrier": 183,
-    "Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis": 48,
-    "Labrador retriever": 208,
-    "Lakeland terrier": 189,
-    "Leonberg": 255,
-    "Lhasa, Lhasa apso": 204,
-    "Loafer": 630,
-    "Madagascar cat, ring-tailed lemur, Lemur catta": 383,
-    "Maltese dog, Maltese terrier, Maltese": 153,
-    "Mexican hairless": 268,
-    "Model T": 661,
-    "Newfoundland, Newfoundland dog": 256,
-    "Norfolk terrier": 185,
-    "Norwegian elkhound, elkhound": 174,
-    "Norwich terrier": 186,
-    "Old English sheepdog, bobtail": 229,
-    "Pekinese, Pekingese, Peke": 154,
-    "Pembroke, Pembroke Welsh corgi": 263,
-    "Persian cat": 283,
-    "Petri dish": 712,
-    "Polaroid camera, Polaroid Land camera": 732,
-    "Pomeranian": 259,
-    "Rhodesian ridgeback": 159,
-    "Rottweiler": 234,
-    "Saint Bernard, St Bernard": 247,
-    "Saluki, gazelle hound": 176,
-    "Samoyed, Samoyede": 258,
-    "Scotch terrier, Scottish terrier, Scottie": 199,
-    "Scottish deerhound, deerhound": 177,
-    "Sealyham terrier, Sealyham": 190,
-    "Shetland sheepdog, Shetland sheep dog, Shetland": 230,
-    "Shih-Tzu": 155,
-    "Siamese cat, Siamese": 284,
-    "Siberian husky": 250,
-    "Staffordshire bullterrier, Staffordshire bull terrier": 179,
-    "Sussex spaniel": 220,
-    "Tibetan mastiff": 244,
-    "Tibetan terrier, chrysanthemum dog": 200,
-    "Walker hound, Walker foxhound": 166,
-    "Weimaraner": 178,
-    "Welsh springer spaniel": 218,
-    "West Highland white terrier": 203,
-    "Windsor tie": 906,
-    "Yorkshire terrier": 187,
-    "abacus": 398,
-    "abaya": 399,
-    "academic gown, academic robe, judge's robe": 400,
-    "accordion, piano accordion, squeeze box": 401,
-    "acorn": 988,
-    "acorn squash": 941,
-    "acoustic guitar": 402,
-    "admiral": 321,
-    "affenpinscher, monkey pinscher, monkey dog": 252,
-    "agama": 42,
-    "agaric": 992,
-    "aircraft carrier, carrier, flattop, attack aircraft carrier": 403,
-    "airliner": 404,
-    "airship, dirigible": 405,
-    "albatross, mollymawk": 146,
-    "alligator lizard": 44,
-    "alp": 970,
-    "altar": 406,
-    "ambulance": 407,
-    "amphibian, amphibious vehicle": 408,
-    "analog clock": 409,
-    "anemone fish": 393,
-    "ant, emmet, pismire": 310,
-    "apiary, bee house": 410,
-    "apron": 411,
-    "armadillo": 363,
-    "artichoke, globe artichoke": 944,
-    "ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin": 412,
-    "assault rifle, assault gun": 413,
-    "axolotl, mud puppy, Ambystoma mexicanum": 29,
-    "baboon": 372,
-    "backpack, back pack, knapsack, packsack, rucksack, haversack": 414,
-    "badger": 362,
-    "bagel, beigel": 931,
-    "bakery, bakeshop, bakehouse": 415,
-    "balance beam, beam": 416,
-    "bald eagle, American eagle, Haliaeetus leucocephalus": 22,
-    "balloon": 417,
-    "ballplayer, baseball player": 981,
-    "ballpoint, ballpoint pen, ballpen, Biro": 418,
-    "banana": 954,
-    "banded gecko": 38,
-    "banjo": 420,
-    "bannister, banister, balustrade, balusters, handrail": 421,
-    "barbell": 422,
-    "barber chair": 423,
-    "barbershop": 424,
-    "barn": 425,
-    "barn spider, Araneus cavaticus": 73,
-    "barometer": 426,
-    "barracouta, snoek": 389,
-    "barrel, cask": 427,
-    "barrow, garden cart, lawn cart, wheelbarrow": 428,
-    "baseball": 429,
-    "basenji": 253,
-    "basketball": 430,
-    "basset, basset hound": 161,
-    "bassinet": 431,
-    "bassoon": 432,
-    "bath towel": 434,
-    "bathing cap, swimming cap": 433,
-    "bathtub, bathing tub, bath, tub": 435,
-    "beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon": 436,
-    "beacon, lighthouse, beacon light, pharos": 437,
-    "beagle": 162,
-    "beaker": 438,
-    "bearskin, busby, shako": 439,
-    "beaver": 337,
-    "bee": 309,
-    "bee eater": 92,
-    "beer bottle": 440,
-    "beer glass": 441,
-    "bell cote, bell cot": 442,
-    "bell pepper": 945,
-    "bib": 443,
-    "bicycle-built-for-two, tandem bicycle, tandem": 444,
-    "bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis": 349,
-    "bikini, two-piece": 445,
-    "binder, ring-binder": 446,
-    "binoculars, field glasses, opera glasses": 447,
-    "birdhouse": 448,
-    "bison": 347,
-    "bittern": 133,
-    "black and gold garden spider, Argiope aurantia": 72,
-    "black grouse": 80,
-    "black stork, Ciconia nigra": 128,
-    "black swan, Cygnus atratus": 100,
-    "black widow, Latrodectus mactans": 75,
-    "black-and-tan coonhound": 165,
-    "black-footed ferret, ferret, Mustela nigripes": 359,
-    "bloodhound, sleuthhound": 163,
-    "bluetick": 164,
-    "boa constrictor, Constrictor constrictor": 61,
-    "boathouse": 449,
-    "bobsled, bobsleigh, bob": 450,
-    "bolete": 997,
-    "bolo tie, bolo, bola tie, bola": 451,
-    "bonnet, poke bonnet": 452,
-    "book jacket, dust cover, dust jacket, dust wrapper": 921,
-    "bookcase": 453,
-    "bookshop, bookstore, bookstall": 454,
-    "borzoi, Russian wolfhound": 169,
-    "bottlecap": 455,
-    "bow": 456,
-    "bow tie, bow-tie, bowtie": 457,
-    "box turtle, box tortoise": 37,
-    "boxer": 242,
-    "brain coral": 109,
-    "brambling, Fringilla montifringilla": 10,
-    "brass, memorial tablet, plaque": 458,
-    "brassiere, bra, bandeau": 459,
-    "breakwater, groin, groyne, mole, bulwark, seawall, jetty": 460,
-    "breastplate, aegis, egis": 461,
-    "briard": 226,
-    "broccoli": 937,
-    "broom": 462,
-    "brown bear, bruin, Ursus arctos": 294,
-    "bubble": 971,
-    "bucket, pail": 463,
-    "buckeye, horse chestnut, conker": 990,
-    "buckle": 464,
-    "bulbul": 16,
-    "bull mastiff": 243,
-    "bullet train, bullet": 466,
-    "bulletproof vest": 465,
-    "bullfrog, Rana catesbeiana": 30,
-    "burrito": 965,
-    "bustard": 138,
-    "butcher shop, meat market": 467,
-    "butternut squash": 942,
-    "cab, hack, taxi, taxicab": 468,
-    "cabbage butterfly": 324,
-    "cairn, cairn terrier": 192,
-    "caldron, cauldron": 469,
-    "can opener, tin opener": 473,
-    "candle, taper, wax light": 470,
-    "cannon": 471,
-    "canoe": 472,
-    "capuchin, ringtail, Cebus capucinus": 378,
-    "car mirror": 475,
-    "car wheel": 479,
-    "carbonara": 959,
-    "cardigan": 474,
-    "cardoon": 946,
-    "carousel, carrousel, merry-go-round, roundabout, whirligig": 476,
-    "carpenter's kit, tool kit": 477,
-    "carton": 478,
-    "cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM": 480,
-    "cassette": 481,
-    "cassette player": 482,
-    "castle": 483,
-    "catamaran": 484,
-    "cauliflower": 938,
-    "cello, violoncello": 486,
-    "cellular telephone, cellular phone, cellphone, cell, mobile phone": 487,
-    "centipede": 79,
-    "chain": 488,
-    "chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour": 490,
-    "chain saw, chainsaw": 491,
-    "chainlink fence": 489,
-    "chambered nautilus, pearly nautilus, nautilus": 117,
-    "cheeseburger": 933,
-    "cheetah, chetah, Acinonyx jubatus": 293,
-    "chest": 492,
-    "chickadee": 19,
-    "chiffonier, commode": 493,
-    "chime, bell, gong": 494,
-    "chimpanzee, chimp, Pan troglodytes": 367,
-    "china cabinet, china closet": 495,
-    "chiton, coat-of-mail shell, sea cradle, polyplacophore": 116,
-    "chocolate sauce, chocolate syrup": 960,
-    "chow, chow chow": 260,
-    "church, church building": 497,
-    "cicada, cicala": 316,
-    "cinema, movie theater, movie theatre, movie house, picture palace": 498,
-    "cleaver, meat cleaver, chopper": 499,
-    "cliff dwelling": 500,
-    "cliff, drop, drop-off": 972,
-    "cloak": 501,
-    "clog, geta, patten, sabot": 502,
-    "clumber, clumber spaniel": 216,
-    "cock": 7,
-    "cocker spaniel, English cocker spaniel, cocker": 219,
-    "cockroach, roach": 314,
-    "cocktail shaker": 503,
-    "coffee mug": 504,
-    "coffeepot": 505,
-    "coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch": 391,
-    "coil, spiral, volute, whorl, helix": 506,
-    "collie": 231,
-    "colobus, colobus monkey": 375,
-    "combination lock": 507,
-    "comic book": 917,
-    "common iguana, iguana, Iguana iguana": 39,
-    "common newt, Triturus vulgaris": 26,
-    "computer keyboard, keypad": 508,
-    "conch": 112,
-    "confectionery, confectionary, candy store": 509,
-    "consomme": 925,
-    "container ship, containership, container vessel": 510,
-    "convertible": 511,
-    "coral fungus": 991,
-    "coral reef": 973,
-    "corkscrew, bottle screw": 512,
-    "corn": 987,
-    "cornet, horn, trumpet, trump": 513,
-    "coucal": 91,
-    "cougar, puma, catamount, mountain lion, painter, panther, Felis concolor": 286,
-    "cowboy boot": 514,
-    "cowboy hat, ten-gallon hat": 515,
-    "coyote, prairie wolf, brush wolf, Canis latrans": 272,
-    "cradle": 516,
-    "crane": 517,
-    "crash helmet": 518,
-    "crate": 519,
-    "crayfish, crawfish, crawdad, crawdaddy": 124,
-    "crib, cot": 520,
-    "cricket": 312,
-    "croquet ball": 522,
-    "crossword puzzle, crossword": 918,
-    "crutch": 523,
-    "cucumber, cuke": 943,
-    "cuirass": 524,
-    "cup": 968,
-    "curly-coated retriever": 206,
-    "custard apple": 956,
-    "daisy": 985,
-    "dalmatian, coach dog, carriage dog": 251,
-    "dam, dike, dyke": 525,
-    "damselfly": 320,
-    "desk": 526,
-    "desktop computer": 527,
-    "dhole, Cuon alpinus": 274,
-    "dial telephone, dial phone": 528,
-    "diamondback, diamondback rattlesnake, Crotalus adamanteus": 67,
-    "diaper, nappy, napkin": 529,
-    "digital clock": 530,
-    "digital watch": 531,
-    "dingo, warrigal, warragal, Canis dingo": 273,
-    "dining table, board": 532,
-    "dishrag, dishcloth": 533,
-    "dishwasher, dish washer, dishwashing machine": 534,
-    "disk brake, disc brake": 535,
-    "dock, dockage, docking facility": 536,
-    "dogsled, dog sled, dog sleigh": 537,
-    "dome": 538,
-    "doormat, welcome mat": 539,
-    "dough": 961,
-    "dowitcher": 142,
-    "dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk": 319,
-    "drake": 97,
-    "drilling platform, offshore rig": 540,
-    "drum, membranophone, tympan": 541,
-    "drumstick": 542,
-    "dugong, Dugong dugon": 149,
-    "dumbbell": 543,
-    "dung beetle": 305,
-    "ear, spike, capitulum": 998,
-    "earthstar": 995,
-    "echidna, spiny anteater, anteater": 102,
-    "eel": 390,
-    "eft": 27,
-    "eggnog": 969,
-    "electric fan, blower": 545,
-    "electric guitar": 546,
-    "electric locomotive": 547,
-    "electric ray, crampfish, numbfish, torpedo": 5,
-    "entertainment center": 548,
-    "envelope": 549,
-    "espresso": 967,
-    "espresso maker": 550,
-    "face powder": 551,
-    "feather boa, boa": 552,
-    "fiddler crab": 120,
-    "fig": 952,
-    "file, file cabinet, filing cabinet": 553,
-    "fire engine, fire truck": 555,
-    "fire screen, fireguard": 556,
-    "fireboat": 554,
-    "flagpole, flagstaff": 557,
-    "flamingo": 130,
-    "flat-coated retriever": 205,
-    "flatworm, platyhelminth": 110,
-    "flute, transverse flute": 558,
-    "fly": 308,
-    "folding chair": 559,
-    "football helmet": 560,
-    "forklift": 561,
-    "fountain": 562,
-    "fountain pen": 563,
-    "four-poster": 564,
-    "fox squirrel, eastern fox squirrel, Sciurus niger": 335,
-    "freight car": 565,
-    "frilled lizard, Chlamydosaurus kingi": 43,
-    "frying pan, frypan, skillet": 567,
-    "fur coat": 568,
-    "gar, garfish, garpike, billfish, Lepisosteus osseus": 395,
-    "garbage truck, dustcart": 569,
-    "garden spider, Aranea diademata": 74,
-    "garter snake, grass snake": 57,
-    "gas pump, gasoline pump, petrol pump, island dispenser": 571,
-    "gasmask, respirator, gas helmet": 570,
-    "gazelle": 353,
-    "geyser": 974,
-    "giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca": 388,
-    "giant schnauzer": 197,
-    "gibbon, Hylobates lar": 368,
-    "go-kart": 573,
-    "goblet": 572,
-    "golden retriever": 207,
-    "goldfinch, Carduelis carduelis": 11,
-    "goldfish, Carassius auratus": 1,
-    "golf ball": 574,
-    "golfcart, golf cart": 575,
-    "gondola": 576,
-    "gong, tam-tam": 577,
-    "goose": 99,
-    "gorilla, Gorilla gorilla": 366,
-    "gown": 578,
-    "grand piano, grand": 579,
-    "grasshopper, hopper": 311,
-    "great grey owl, great gray owl, Strix nebulosa": 24,
-    "great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias": 2,
-    "green lizard, Lacerta viridis": 46,
-    "green mamba": 64,
-    "green snake, grass snake": 55,
-    "greenhouse, nursery, glasshouse": 580,
-    "grey fox, gray fox, Urocyon cinereoargenteus": 280,
-    "grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus": 147,
-    "grille, radiator grille": 581,
-    "grocery store, grocery, food market, market": 582,
-    "groenendael": 224,
-    "groom, bridegroom": 982,
-    "ground beetle, carabid beetle": 302,
-    "guacamole": 924,
-    "guenon, guenon monkey": 370,
-    "guillotine": 583,
-    "guinea pig, Cavia cobaya": 338,
-    "gyromitra": 993,
-    "hair slide": 584,
-    "hair spray": 585,
-    "half track": 586,
-    "hammer": 587,
-    "hammerhead, hammerhead shark": 4,
-    "hamper": 588,
-    "hamster": 333,
-    "hand blower, blow dryer, blow drier, hair dryer, hair drier": 589,
-    "hand-held computer, hand-held microcomputer": 590,
-    "handkerchief, hankie, hanky, hankey": 591,
-    "hard disc, hard disk, fixed disk": 592,
-    "hare": 331,
-    "harmonica, mouth organ, harp, mouth harp": 593,
-    "harp": 594,
-    "hartebeest": 351,
-    "harvester, reaper": 595,
-    "harvestman, daddy longlegs, Phalangium opilio": 70,
-    "hatchet": 596,
-    "hay": 958,
-    "head cabbage": 936,
-    "hen": 8,
-    "hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa": 996,
-    "hermit crab": 125,
-    "hip, rose hip, rosehip": 989,
-    "hippopotamus, hippo, river horse, Hippopotamus amphibius": 344,
-    "hog, pig, grunter, squealer, Sus scrofa": 341,
-    "hognose snake, puff adder, sand viper": 54,
-    "holster": 597,
-    "home theater, home theatre": 598,
-    "honeycomb": 599,
-    "hook, claw": 600,
-    "hoopskirt, crinoline": 601,
-    "horizontal bar, high bar": 602,
-    "hornbill": 93,
-    "horned viper, cerastes, sand viper, horned asp, Cerastes cornutus": 66,
-    "horse cart, horse-cart": 603,
-    "hot pot, hotpot": 926,
-    "hotdog, hot dog, red hot": 934,
-    "hourglass": 604,
-    "house finch, linnet, Carpodacus mexicanus": 12,
-    "howler monkey, howler": 379,
-    "hummingbird": 94,
-    "hyena, hyaena": 276,
-    "iPod": 605,
-    "ibex, Capra ibex": 350,
-    "ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus": 296,
-    "ice cream, icecream": 928,
-    "ice lolly, lolly, lollipop, popsicle": 929,
-    "impala, Aepyceros melampus": 352,
-    "indigo bunting, indigo finch, indigo bird, Passerina cyanea": 14,
-    "indri, indris, Indri indri, Indri brevicaudatus": 384,
-    "iron, smoothing iron": 606,
-    "isopod": 126,
-    "jacamar": 95,
-    "jack-o'-lantern": 607,
-    "jackfruit, jak, jack": 955,
-    "jaguar, panther, Panthera onca, Felis onca": 290,
-    "jay": 17,
-    "jean, blue jean, denim": 608,
-    "jeep, landrover": 609,
-    "jellyfish": 107,
-    "jersey, T-shirt, tee shirt": 610,
-    "jigsaw puzzle": 611,
-    "jinrikisha, ricksha, rickshaw": 612,
-    "joystick": 613,
-    "junco, snowbird": 13,
-    "keeshond": 261,
-    "kelpie": 227,
-    "killer whale, killer, orca, grampus, sea wolf, Orcinus orca": 148,
-    "kimono": 614,
-    "king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica": 121,
-    "king penguin, Aptenodytes patagonica": 145,
-    "king snake, kingsnake": 56,
-    "kit fox, Vulpes macrotis": 278,
-    "kite": 21,
-    "knee pad": 615,
-    "knot": 616,
-    "koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus": 105,
-    "komondor": 228,
-    "kuvasz": 222,
-    "lab coat, laboratory coat": 617,
-    "lacewing, lacewing fly": 318,
-    "ladle": 618,
-    "ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle": 301,
-    "lakeside, lakeshore": 975,
-    "lampshade, lamp shade": 619,
-    "langur": 374,
-    "laptop, laptop computer": 620,
-    "lawn mower, mower": 621,
-    "leaf beetle, chrysomelid": 304,
-    "leafhopper": 317,
-    "leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea": 34,
-    "lemon": 951,
-    "lens cap, lens cover": 622,
-    "leopard, Panthera pardus": 288,
-    "lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens": 387,
-    "letter opener, paper knife, paperknife": 623,
-    "library": 624,
-    "lifeboat": 625,
-    "lighter, light, igniter, ignitor": 626,
-    "limousine, limo": 627,
-    "limpkin, Aramus pictus": 135,
-    "liner, ocean liner": 628,
-    "lion, king of beasts, Panthera leo": 291,
-    "lionfish": 396,
-    "lipstick, lip rouge": 629,
-    "little blue heron, Egretta caerulea": 131,
-    "llama": 355,
-    "loggerhead, loggerhead turtle, Caretta caretta": 33,
-    "long-horned beetle, longicorn, longicorn beetle": 303,
-    "lorikeet": 90,
-    "lotion": 631,
-    "loudspeaker, speaker, speaker unit, loudspeaker system, speaker system": 632,
-    "loupe, jeweler's loupe": 633,
-    "lumbermill, sawmill": 634,
-    "lycaenid, lycaenid butterfly": 326,
-    "lynx, catamount": 287,
-    "macaque": 373,
-    "macaw": 88,
-    "magnetic compass": 635,
-    "magpie": 18,
-    "mailbag, postbag": 636,
-    "mailbox, letter box": 637,
-    "maillot": 638,
-    "maillot, tank suit": 639,
-    "malamute, malemute, Alaskan malamute": 249,
-    "malinois": 225,
-    "manhole cover": 640,
-    "mantis, mantid": 315,
-    "maraca": 641,
-    "marimba, xylophone": 642,
-    "marmoset": 377,
-    "marmot": 336,
-    "mashed potato": 935,
-    "mask": 643,
-    "matchstick": 644,
-    "maypole": 645,
-    "maze, labyrinth": 646,
-    "measuring cup": 647,
-    "meat loaf, meatloaf": 962,
-    "medicine chest, medicine cabinet": 648,
-    "meerkat, mierkat": 299,
-    "megalith, megalithic structure": 649,
-    "menu": 922,
-    "microphone, mike": 650,
-    "microwave, microwave oven": 651,
-    "military uniform": 652,
-    "milk can": 653,
-    "miniature pinscher": 237,
-    "miniature poodle": 266,
-    "miniature schnauzer": 196,
-    "minibus": 654,
-    "miniskirt, mini": 655,
-    "minivan": 656,
-    "mink": 357,
-    "missile": 657,
-    "mitten": 658,
-    "mixing bowl": 659,
-    "mobile home, manufactured home": 660,
-    "modem": 662,
-    "monarch, monarch butterfly, milkweed butterfly, Danaus plexippus": 323,
-    "monastery": 663,
-    "mongoose": 298,
-    "monitor": 664,
-    "moped": 665,
-    "mortar": 666,
-    "mortarboard": 667,
-    "mosque": 668,
-    "mosquito net": 669,
-    "motor scooter, scooter": 670,
-    "mountain bike, all-terrain bike, off-roader": 671,
-    "mountain tent": 672,
-    "mouse, computer mouse": 673,
-    "mousetrap": 674,
-    "moving van": 675,
-    "mud turtle": 35,
-    "mushroom": 947,
-    "muzzle": 676,
-    "nail": 677,
-    "neck brace": 678,
-    "necklace": 679,
-    "nematode, nematode worm, roundworm": 111,
-    "night snake, Hypsiglena torquata": 60,
-    "nipple": 680,
-    "notebook, notebook computer": 681,
-    "obelisk": 682,
-    "oboe, hautboy, hautbois": 683,
-    "ocarina, sweet potato": 684,
-    "odometer, hodometer, mileometer, milometer": 685,
-    "oil filter": 686,
-    "orange": 950,
-    "orangutan, orang, orangutang, Pongo pygmaeus": 365,
-    "organ, pipe organ": 687,
-    "oscilloscope, scope, cathode-ray oscilloscope, CRO": 688,
-    "ostrich, Struthio camelus": 9,
-    "otter": 360,
-    "otterhound, otter hound": 175,
-    "overskirt": 689,
-    "ox": 345,
-    "oxcart": 690,
-    "oxygen mask": 691,
-    "oystercatcher, oyster catcher": 143,
-    "packet": 692,
-    "paddle, boat paddle": 693,
-    "paddlewheel, paddle wheel": 694,
-    "padlock": 695,
-    "paintbrush": 696,
-    "pajama, pyjama, pj's, jammies": 697,
-    "palace": 698,
-    "panpipe, pandean pipe, syrinx": 699,
-    "paper towel": 700,
-    "papillon": 157,
-    "parachute, chute": 701,
-    "parallel bars, bars": 702,
-    "park bench": 703,
-    "parking meter": 704,
-    "partridge": 86,
-    "passenger car, coach, carriage": 705,
-    "patas, hussar monkey, Erythrocebus patas": 371,
-    "patio, terrace": 706,
-    "pay-phone, pay-station": 707,
-    "peacock": 84,
-    "pedestal, plinth, footstall": 708,
-    "pelican": 144,
-    "pencil box, pencil case": 709,
-    "pencil sharpener": 710,
-    "perfume, essence": 711,
-    "photocopier": 713,
-    "pick, plectrum, plectron": 714,
-    "pickelhaube": 715,
-    "picket fence, paling": 716,
-    "pickup, pickup truck": 717,
-    "pier": 718,
-    "piggy bank, penny bank": 719,
-    "pill bottle": 720,
-    "pillow": 721,
-    "pineapple, ananas": 953,
-    "ping-pong ball": 722,
-    "pinwheel": 723,
-    "pirate, pirate ship": 724,
-    "pitcher, ewer": 725,
-    "pizza, pizza pie": 963,
-    "plane, carpenter's plane, woodworking plane": 726,
-    "planetarium": 727,
-    "plastic bag": 728,
-    "plate": 923,
-    "plate rack": 729,
-    "platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus": 103,
-    "plow, plough": 730,
-    "plunger, plumber's helper": 731,
-    "pole": 733,
-    "polecat, fitch, foulmart, foumart, Mustela putorius": 358,
-    "police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria": 734,
-    "pomegranate": 957,
-    "poncho": 735,
-    "pool table, billiard table, snooker table": 736,
-    "pop bottle, soda bottle": 737,
-    "porcupine, hedgehog": 334,
-    "pot, flowerpot": 738,
-    "potpie": 964,
-    "potter's wheel": 739,
-    "power drill": 740,
-    "prairie chicken, prairie grouse, prairie fowl": 83,
-    "prayer rug, prayer mat": 741,
-    "pretzel": 932,
-    "printer": 742,
-    "prison, prison house": 743,
-    "proboscis monkey, Nasalis larvatus": 376,
-    "projectile, missile": 744,
-    "projector": 745,
-    "promontory, headland, head, foreland": 976,
-    "ptarmigan": 81,
-    "puck, hockey puck": 746,
-    "puffer, pufferfish, blowfish, globefish": 397,
-    "pug, pug-dog": 254,
-    "punching bag, punch bag, punching ball, punchball": 747,
-    "purse": 748,
-    "quail": 85,
-    "quill, quill pen": 749,
-    "quilt, comforter, comfort, puff": 750,
-    "racer, race car, racing car": 751,
-    "racket, racquet": 752,
-    "radiator": 753,
-    "radio telescope, radio reflector": 755,
-    "radio, wireless": 754,
-    "rain barrel": 756,
-    "ram, tup": 348,
-    "rapeseed": 984,
-    "recreational vehicle, RV, R.V.": 757,
-    "red fox, Vulpes vulpes": 277,
-    "red wine": 966,
-    "red wolf, maned wolf, Canis rufus, Canis niger": 271,
-    "red-backed sandpiper, dunlin, Erolia alpina": 140,
-    "red-breasted merganser, Mergus serrator": 98,
-    "redbone": 168,
-    "redshank, Tringa totanus": 141,
-    "reel": 758,
-    "reflex camera": 759,
-    "refrigerator, icebox": 760,
-    "remote control, remote": 761,
-    "restaurant, eating house, eating place, eatery": 762,
-    "revolver, six-gun, six-shooter": 763,
-    "rhinoceros beetle": 306,
-    "rifle": 764,
-    "ringlet, ringlet butterfly": 322,
-    "ringneck snake, ring-necked snake, ring snake": 53,
-    "robin, American robin, Turdus migratorius": 15,
-    "rock beauty, Holocanthus tricolor": 392,
-    "rock crab, Cancer irroratus": 119,
-    "rock python, rock snake, Python sebae": 62,
-    "rocking chair, rocker": 765,
-    "rotisserie": 766,
-    "rubber eraser, rubber, pencil eraser": 767,
-    "ruddy turnstone, Arenaria interpres": 139,
-    "ruffed grouse, partridge, Bonasa umbellus": 82,
-    "rugby ball": 768,
-    "rule, ruler": 769,
-    "running shoe": 770,
-    "safe": 771,
-    "safety pin": 772,
-    "saltshaker, salt shaker": 773,
-    "sandal": 774,
-    "sandbar, sand bar": 977,
-    "sarong": 775,
-    "sax, saxophone": 776,
-    "scabbard": 777,
-    "scale, weighing machine": 778,
-    "schipperke": 223,
-    "school bus": 779,
-    "schooner": 780,
-    "scoreboard": 781,
-    "scorpion": 71,
-    "screen, CRT screen": 782,
-    "screw": 783,
-    "screwdriver": 784,
-    "scuba diver": 983,
-    "sea anemone, anemone": 108,
-    "sea cucumber, holothurian": 329,
-    "sea lion": 150,
-    "sea slug, nudibranch": 115,
-    "sea snake": 65,
-    "sea urchin": 328,
-    "seashore, coast, seacoast, sea-coast": 978,
-    "seat belt, seatbelt": 785,
-    "sewing machine": 786,
-    "shield, buckler": 787,
-    "shoe shop, shoe-shop, shoe store": 788,
-    "shoji": 789,
-    "shopping basket": 790,
-    "shopping cart": 791,
-    "shovel": 792,
-    "shower cap": 793,
-    "shower curtain": 794,
-    "siamang, Hylobates syndactylus, Symphalangus syndactylus": 369,
-    "sidewinder, horned rattlesnake, Crotalus cerastes": 68,
-    "silky terrier, Sydney silky": 201,
-    "ski": 795,
-    "ski mask": 796,
-    "skunk, polecat, wood pussy": 361,
-    "sleeping bag": 797,
-    "slide rule, slipstick": 798,
-    "sliding door": 799,
-    "slot, one-armed bandit": 800,
-    "sloth bear, Melursus ursinus, Ursus ursinus": 297,
-    "slug": 114,
-    "snail": 113,
-    "snorkel": 801,
-    "snow leopard, ounce, Panthera uncia": 289,
-    "snowmobile": 802,
-    "snowplow, snowplough": 803,
-    "soap dispenser": 804,
-    "soccer ball": 805,
-    "sock": 806,
-    "soft-coated wheaten terrier": 202,
-    "solar dish, solar collector, solar furnace": 807,
-    "sombrero": 808,
-    "sorrel": 339,
-    "soup bowl": 809,
-    "space bar": 810,
-    "space heater": 811,
-    "space shuttle": 812,
-    "spaghetti squash": 940,
-    "spatula": 813,
-    "speedboat": 814,
-    "spider monkey, Ateles geoffroyi": 381,
-    "spider web, spider's web": 815,
-    "spindle": 816,
-    "spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish": 123,
-    "spoonbill": 129,
-    "sports car, sport car": 817,
-    "spotlight, spot": 818,
-    "spotted salamander, Ambystoma maculatum": 28,
-    "squirrel monkey, Saimiri sciureus": 382,
-    "stage": 819,
-    "standard poodle": 267,
-    "standard schnauzer": 198,
-    "starfish, sea star": 327,
-    "steam locomotive": 820,
-    "steel arch bridge": 821,
-    "steel drum": 822,
-    "stethoscope": 823,
-    "stingray": 6,
-    "stinkhorn, carrion fungus": 994,
-    "stole": 824,
-    "stone wall": 825,
-    "stopwatch, stop watch": 826,
-    "stove": 827,
-    "strainer": 828,
-    "strawberry": 949,
-    "street sign": 919,
-    "streetcar, tram, tramcar, trolley, trolley car": 829,
-    "stretcher": 830,
-    "studio couch, day bed": 831,
-    "stupa, tope": 832,
-    "sturgeon": 394,
-    "submarine, pigboat, sub, U-boat": 833,
-    "suit, suit of clothes": 834,
-    "sulphur butterfly, sulfur butterfly": 325,
-    "sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita": 89,
-    "sundial": 835,
-    "sunglass": 836,
-    "sunglasses, dark glasses, shades": 837,
-    "sunscreen, sunblock, sun blocker": 838,
-    "suspension bridge": 839,
-    "swab, swob, mop": 840,
-    "sweatshirt": 841,
-    "swimming trunks, bathing trunks": 842,
-    "swing": 843,
-    "switch, electric switch, electrical switch": 844,
-    "syringe": 845,
-    "tabby, tabby cat": 281,
-    "table lamp": 846,
-    "tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui": 32,
-    "tank, army tank, armored combat vehicle, armoured combat vehicle": 847,
-    "tape player": 848,
-    "tarantula": 76,
-    "teapot": 849,
-    "teddy, teddy bear": 850,
-    "television, television system": 851,
-    "tench, Tinca tinca": 0,
-    "tennis ball": 852,
-    "terrapin": 36,
-    "thatch, thatched roof": 853,
-    "theater curtain, theatre curtain": 854,
-    "thimble": 855,
-    "three-toed sloth, ai, Bradypus tridactylus": 364,
-    "thresher, thrasher, threshing machine": 856,
-    "throne": 857,
-    "thunder snake, worm snake, Carphophis amoenus": 52,
-    "tick": 78,
-    "tiger beetle": 300,
-    "tiger cat": 282,
-    "tiger shark, Galeocerdo cuvieri": 3,
-    "tiger, Panthera tigris": 292,
-    "tile roof": 858,
-    "timber wolf, grey wolf, gray wolf, Canis lupus": 269,
-    "titi, titi monkey": 380,
-    "toaster": 859,
-    "tobacco shop, tobacconist shop, tobacconist": 860,
-    "toilet seat": 861,
-    "toilet tissue, toilet paper, bathroom tissue": 999,
-    "torch": 862,
-    "totem pole": 863,
-    "toucan": 96,
-    "tow truck, tow car, wrecker": 864,
-    "toy poodle": 265,
-    "toy terrier": 158,
-    "toyshop": 865,
-    "tractor": 866,
-    "traffic light, traffic signal, stoplight": 920,
-    "trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi": 867,
-    "tray": 868,
-    "tree frog, tree-frog": 31,
-    "trench coat": 869,
-    "triceratops": 51,
-    "tricycle, trike, velocipede": 870,
-    "trifle": 927,
-    "trilobite": 69,
-    "trimaran": 871,
-    "tripod": 872,
-    "triumphal arch": 873,
-    "trolleybus, trolley coach, trackless trolley": 874,
-    "trombone": 875,
-    "tub, vat": 876,
-    "turnstile": 877,
-    "tusker": 101,
-    "typewriter keyboard": 878,
-    "umbrella": 879,
-    "unicycle, monocycle": 880,
-    "upright, upright piano": 881,
-    "vacuum, vacuum cleaner": 882,
-    "valley, vale": 979,
-    "vase": 883,
-    "vault": 884,
-    "velvet": 885,
-    "vending machine": 886,
-    "vestment": 887,
-    "viaduct": 888,
-    "vine snake": 59,
-    "violin, fiddle": 889,
-    "vizsla, Hungarian pointer": 211,
-    "volcano": 980,
-    "volleyball": 890,
-    "vulture": 23,
-    "waffle iron": 891,
-    "walking stick, walkingstick, stick insect": 313,
-    "wall clock": 892,
-    "wallaby, brush kangaroo": 104,
-    "wallet, billfold, notecase, pocketbook": 893,
-    "wardrobe, closet, press": 894,
-    "warplane, military plane": 895,
-    "warthog": 343,
-    "washbasin, handbasin, washbowl, lavabo, wash-hand basin": 896,
-    "washer, automatic washer, washing machine": 897,
-    "water bottle": 898,
-    "water buffalo, water ox, Asiatic buffalo, Bubalus bubalis": 346,
-    "water jug": 899,
-    "water ouzel, dipper": 20,
-    "water snake": 58,
-    "water tower": 900,
-    "weasel": 356,
-    "web site, website, internet site, site": 916,
-    "weevil": 307,
-    "whippet": 172,
-    "whiptail, whiptail lizard": 41,
-    "whiskey jug": 901,
-    "whistle": 902,
-    "white stork, Ciconia ciconia": 127,
-    "white wolf, Arctic wolf, Canis lupus tundrarum": 270,
-    "wig": 903,
-    "wild boar, boar, Sus scrofa": 342,
-    "window screen": 904,
-    "window shade": 905,
-    "wine bottle": 907,
-    "wing": 908,
-    "wire-haired fox terrier": 188,
-    "wok": 909,
-    "wolf spider, hunting spider": 77,
-    "wombat": 106,
-    "wood rabbit, cottontail, cottontail rabbit": 330,
-    "wooden spoon": 910,
-    "wool, woolen, woollen": 911,
-    "worm fence, snake fence, snake-rail fence, Virginia fence": 912,
-    "wreck": 913,
-    "yawl": 914,
-    "yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum": 986,
-    "yurt": 915,
-    "zebra": 340,
-    "zucchini, courgette": 939
-  },
-  "layer_norm_eps": 1e-12,
-  "model_type": "vit",
-  "num_attention_heads": 12,
-  "num_channels": 3,
-  "num_hidden_layers": 12,
-  "patch_size": 16,
-  "qkv_bias": true,
-  "transformers_version": "4.13.0.dev0"
-}
diff --git a/models/image_recognition/tensorflow/vision_transformer/training/requirements.txt b/models/image_recognition/tensorflow/vision_transformer/training/requirements.txt
deleted file mode 100644
index 576a4962c..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/training/requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-transformers==4.30.0 # fixed for Snyk scans to avoid vulnerability
-numpy==1.23.5
-tensorflow==2.12.0
diff --git a/models/image_recognition/tensorflow/vision_transformer/training/vit_fine_tune.py b/models/image_recognition/tensorflow/vision_transformer/training/vit_fine_tune.py
deleted file mode 100644
index c5a85fcb4..000000000
--- a/models/image_recognition/tensorflow/vision_transformer/training/vit_fine_tune.py
+++ /dev/null
@@ -1,339 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-
-import time
-import numpy as np
-
-import tensorflow as tf
-from transformers import TFViTForImageClassification
-from argparse import ArgumentParser
-
-print("TF version:", tf.__version__)
-
-def get_arg_parser():
-    arg_parser = ArgumentParser(description='Parse args')
-    arg_parser.add_argument(
-        '-d', '--data-location',
-        help='Specify the location of the training data',
-        dest='data_location', required=True
-    )
-    arg_parser.add_argument(
-        '-p', '--precision',
-        help='Specify the datatyp precision : fp32 / bfloat16 / fp16',
-        dest='precision', default='fp32'
-    )    
-    arg_parser.add_argument(
-        '-s', '--steps',
-        help='Specify the number of steps for training',
-        type=int, dest='steps',
-        default=30000
-    )
-    arg_parser.add_argument(
-        '-b', '--batch-size',
-        help='Batch size for training',
-        type=int, dest='batch_size',
-        default=512
-    )
-    arg_parser.add_argument(
-        '-i', '--init-checkpoint',
-        help='Specify the location of the output directory for logs and saved model',
-        dest='init_checkpoint', required=True
-    )
-    
-    arg_parser.add_argument(
-        '-o', '--model-dir',
-        help='Specify the location of the output directory for logs and saved model',
-        dest='model_dir', default='/tmp/vit-finetuned-model'
-    )
-    return arg_parser
-
-parser = get_arg_parser()
-args = parser.parse_args()
-
-
-model_name = "vit_b32_imagenet21k"
-
-model_handle_map = {
-  "vit_b32_imagenet21k": args.init_checkpoint
-}
-
-model_image_size_map = {
-  "vit_b32_imagenet21k": 224,
-}
-
-model_handle = model_handle_map.get(model_name)
-pixels = model_image_size_map.get(model_name, 224)
-
-print(f"Selected model: {model_name} : {model_handle}")
-
-IMAGE_SIZE = (pixels, pixels)
-print(f"Input size {IMAGE_SIZE}")
-
-BATCH_SIZE = args.batch_size
-
-############ Using ImageNet tf-records############
-### reference : https://keras.io/examples/keras_recipes/tfrecord/
-#######
-
-TRAINING_FILENAMES = tf.io.gfile.glob(args.data_location + "/train-*")
-VALID_FILENAMES = tf.io.gfile.glob(args.data_location + "/validation-*")
-
-print("Train TFRecord Files:", len(TRAINING_FILENAMES))
-print("Validation TFRecord Files:", len(VALID_FILENAMES))
-
-#decode imagenet data
-#The images have to be converted to tensors so that it will be a valid input in our model. As images utilize an RBG scale, we specify 3 channels.
-
-def decode_image(image):
-    image = tf.image.decode_jpeg(image, channels=3)
-    image = tf.image.resize(image, IMAGE_SIZE, method=tf.image.ResizeMethod.BILINEAR)
-    return image
-
-def _parse_example_proto(example_serialized):
-  """Parses an Example proto containing a training example of an image.
-  The output of the build_image_data.py image preprocessing script is a dataset
-  containing serialized Example protocol buffers. Each Example proto contains
-  the following fields (values are included as examples):
-    image/height: 462
-    image/width: 581
-    image/colorspace: 'RGB'
-    image/channels: 3
-    image/class/label: 615
-    image/class/synset: 'n03623198'
-    image/class/text: 'knee pad'
-    image/object/bbox/xmin: 0.1
-    image/object/bbox/xmax: 0.9
-    image/object/bbox/ymin: 0.2
-    image/object/bbox/ymax: 0.6
-    image/object/bbox/label: 615
-    image/format: 'JPEG'
-    image/filename: 'ILSVRC2012_val_00041207.JPEG'
-    image/encoded: <JPEG encoded string>
-  Args:
-    example_serialized: scalar Tensor tf.string containing a serialized
-      Example protocol buffer.
-  Returns:
-    image_buffer: Tensor tf.string containing the contents of a JPEG file.
-    label: Tensor tf.int32 containing the label.
-    bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]
-      where each coordinate is [0, 1) and the coordinates are arranged as
-      [ymin, xmin, ymax, xmax].
-  """
-  # Dense features in Example proto.
-  feature_map = {
-      'image/encoded': tf.io.FixedLenFeature([], dtype=tf.string,
-                                          default_value=''),
-      'image/class/label': tf.io.FixedLenFeature([], dtype=tf.int64,
-                                              default_value=-1),
-      'image/class/text': tf.io.FixedLenFeature([], dtype=tf.string,
-                                             default_value=''),
-  }
-  sparse_float32 = tf.io.VarLenFeature(dtype=tf.float32)
-  # Sparse features in Example proto.
-  feature_map.update(
-      {k: sparse_float32 for k in ['image/object/bbox/xmin',
-                                   'image/object/bbox/ymin',
-                                   'image/object/bbox/xmax',
-                                   'image/object/bbox/ymax']})
-
-  features = tf.io.parse_single_example(serialized=example_serialized, features=feature_map)
-  img = decode_image(features['image/encoded'])
-  label = tf.cast(features['image/class/label'], dtype=tf.int32)
-  label = tf.subtract(label, 1)
-
-  return img, label
-
-
-def load_dataset(filenames, labeled=True):
-    ignore_order = tf.data.Options()
-    ignore_order.experimental_deterministic = False  # disable order, increase speed
-    dataset = tf.data.TFRecordDataset(
-        filenames
-    )  # automatically interleaves reads from multiple files
-
-    dataset = dataset.with_options(
-        ignore_order
-    )  # uses data as soon as it streams in, rather than in its original order
-    dataset = dataset.map(_parse_example_proto, num_parallel_calls=tf.data.AUTOTUNE)
-    # returns a dataset of (image, label) pairs if labeled=True or just images if labeled=False
-    return dataset
-
-def get_dataset(filenames, labeled=True):
-    dataset = load_dataset(filenames, labeled=labeled)
-    dataset = dataset.shuffle(2048)
-    dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)
-    return dataset
-
-train_ds_total = get_dataset(TRAINING_FILENAMES)
-NUM_TRAIN_SAMPLES = 1281167
-num_train_split = int(0.99 * NUM_TRAIN_SAMPLES)
-num_val_split = int(0.01 * NUM_TRAIN_SAMPLES)
-test_ds = get_dataset(VALID_FILENAMES)
-
-#Minval split 99%train - 1%validation
-train_ds = train_ds_total.take(num_train_split)
-val_ds = train_ds_total.skip(num_train_split).take(num_val_split)
-
-def preprocess_data(image, train:bool):
-    # For training, do a random crop and horizontal flip.
-    if train:
-        channels = image.shape[-1]
-        begin, size, _ = tf.image.sample_distorted_bounding_box(
-                tf.shape(image),
-                tf.zeros([0, 0, 4], tf.float32),
-                area_range=(0.05, 1.0),
-                min_object_covered=0,
-                use_image_if_no_bounding_boxes=True,
-        )
-        image = tf.slice(image, begin, size)
-        image.set_shape([None, None, channels])
-        image = tf.image.resize(image, IMAGE_SIZE)
-        if tf.random.uniform(shape=[]) > 0.5:
-            image = tf.image.flip_left_right(image)
-    else:
-        image = tf.image.resize(image, IMAGE_SIZE)
-    image = (image - 127.5) / 127.5
-    image =  tf.transpose(image, perm=[2,0,1])
-    return image
-
-train_ds = train_ds.map(lambda images, labels:
-                        (preprocess_data(images, True), labels)).batch(BATCH_SIZE, drop_remainder=True)
-val_ds = val_ds.map(lambda images, labels:
-                    (preprocess_data(images, False), labels)).batch(BATCH_SIZE, drop_remainder=True)
-test_ds = test_ds.map(lambda images, labels:
-                    (preprocess_data(images, False), labels)).batch(BATCH_SIZE, drop_remainder=True)
-
-do_fine_tuning = True
-
-# Load the pre-trained Vision Transformer model
-model = TFViTForImageClassification.from_pretrained(model_handle)
-
-if not do_fine_tuning:
-    for layer in model.layers[:-1]:
-        layer.trainable = False
-
-model.summary()
-
-#Cosine Learning rate scheduler
-
-class WarmUpCosine(tf.keras.optimizers.schedules.LearningRateSchedule):
-    def __init__(
-        self, learning_rate_base, total_steps, warmup_learning_rate, warmup_steps
-    ):
-        super(WarmUpCosine, self).__init__()
-
-        self.learning_rate_base = learning_rate_base
-        self.total_steps = total_steps
-        self.warmup_learning_rate = warmup_learning_rate
-        self.warmup_steps = warmup_steps
-        self.pi = tf.constant(np.pi)
-
-    def __call__(self, step):
-        if self.total_steps < self.warmup_steps:
-            raise ValueError("Total_steps must be larger or equal to warmup_steps.")
-        learning_rate = (
-            0.5
-            * self.learning_rate_base
-            * (
-                1
-                + tf.cos(
-                    self.pi
-                    * (tf.cast(step, tf.float32) - self.warmup_steps)
-                    / float(self.total_steps - self.warmup_steps)
-                )
-            )
-        )
-
-        if self.warmup_steps > 0:
-            if self.learning_rate_base < self.warmup_learning_rate:
-                raise ValueError(
-                    "Learning_rate_base must be larger or equal to "
-                    "warmup_learning_rate."
-                )
-            slope = (
-                self.learning_rate_base - self.warmup_learning_rate
-            ) / self.warmup_steps
-            warmup_rate = slope * tf.cast(step, tf.float32) + self.warmup_learning_rate
-            learning_rate = tf.where(
-                step < self.warmup_steps, warmup_rate, learning_rate
-            )
-        return tf.where(
-            step > self.total_steps, 0.0, learning_rate, name="learning_rate"
-        )
-    def get_config(self):
-        config = {'d_model': self.d_model, 'warmup_steps': self.warmup_steps}
-        return config
-
-TOTAL_STEPS = args.steps
-WARMUP_STEPS = 50
-INIT_LR = 0.03
-WAMRUP_LR = 0.01
-EPOCHS = int(TOTAL_STEPS / (num_train_split/BATCH_SIZE))
-
-scheduled_lrs = WarmUpCosine(
-    learning_rate_base=INIT_LR,
-    total_steps=TOTAL_STEPS,
-    warmup_learning_rate=WAMRUP_LR,
-    warmup_steps=WARMUP_STEPS,
-)
-
-if args.precision == 'bfloat16':
-    tf.config.optimizer.set_experimental_options({'auto_mixed_precision_onednn_bfloat16':True})
-elif args.precision == 'fp16':
-    tf.config.optimizer.set_experimental_options({'auto_mixed_precision':True})
-
-optimizer = tf.keras.optimizers.SGD(scheduled_lrs, momentum=0.9, clipnorm=1.0)
-loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
-model.compile(loss=loss, optimizer=optimizer, metrics=["accuracy"])
-
-class TimeHistory(tf.keras.callbacks.Callback):
-    def on_train_begin(self, logs={}):
-        self.times = []
-        self.throughput = []
-        print("<========= TRAINING START ========>")
-
-    def on_batch_begin(self, batch, logs={}):
-        self.epoch_time_start = time.time()
-
-    def on_batch_end(self, batch, logs={}):
-        total_time = time.time() - self.epoch_time_start
-        self.times.append(total_time)
-        self.throughput.append(BATCH_SIZE/total_time)
-
-    def on_train_end(self, logs={}):
-        print("<========= TRAINING END =========>")
-
-    def on_epoch_begin(self, epoch, logs={}):
-        print("Start epoch {} of training".format(epoch))
-
-time_callback = TimeHistory()
-
-hist = model.fit(
-    train_ds,
-    epochs=EPOCHS,
-    validation_data=val_ds, callbacks=[time_callback]).history
-
-
-avg_throughput = sum(time_callback.throughput)/len(time_callback.throughput)
-print("Avg Throughput: " + str(avg_throughput) + " imgs/sec")
-
-# Evaluate the model
-print("Evaluating trained model for accuracy with test data")
-test_loss, test_acc = model.evaluate(test_ds)
-print('Imagenet Validation set : Test accuracy:', test_acc)
diff --git a/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_accuracy.py b/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_accuracy.py
index 702264453..d31268179 100644
--- a/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_accuracy.py
+++ b/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_accuracy.py
@@ -69,6 +69,9 @@ def __init__(self):
                                 help='For accuracy measurement only.',
                                 dest='accuracy_only', action='store_true')
         arg_parser.add_argument("--batch-size", type=int, default=1)
+        arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
+                                help='enable OneDNN Graph',
+                                action='store_true')
 
         self.args = arg_parser.parse_args()
         print(self.args)
@@ -93,6 +96,25 @@ def run(self):
         config.intra_op_parallelism_threads=self.args.num_intra_threads
         config.inter_op_parallelism_threads=self.args.num_inter_threads
 
+        if self.args.onednn_graph:
+            import intel_extension_for_tensorflow as itex
+            auto_mixed_precision_options = itex.AutoMixedPrecisionOptions()
+            auto_mixed_precision_options.data_type = itex.BFLOAT16
+
+            auto_mixed_precision_options.allowlist_add= "Rsqrt,SquaredDifference,Mean"
+            auto_mixed_precision_options.denylist_remove = "Mean"
+            auto_mixed_precision_options.denylist_add = "QuantizeV2,Dequantize"
+
+            graph_options = itex.GraphOptions(auto_mixed_precision_options=auto_mixed_precision_options)
+            graph_options.auto_mixed_precision = itex.ON
+
+            config = itex.ConfigProto(graph_options=graph_options)
+            try:
+                itex.set_backend("cpu", config)
+            except TypeError:
+                itex.set_config(config)
+            config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
+
         sess = tf.compat.v1.Session(graph=graph, config=config)
         if (self.args.accuracy_only):
             print("Inference with real data")
diff --git a/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_performance.py b/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_performance.py
index d989dff56..95ee52d7d 100644
--- a/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_performance.py
+++ b/models/image_segmentation/tensorflow/3d_unet_mlperf/inference/int8/brats/run_performance.py
@@ -64,6 +64,9 @@ def __init__(self):
         arg_parser.add_argument("--steps", type=int, default=50,
                                 help="number of steps")
         arg_parser.add_argument("--batch-size", type=int, default=1)
+        arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
+                                help='enable OneDNN Graph',
+                                action='store_true')
 
         self.args = arg_parser.parse_args()
         print (self.args)
@@ -96,6 +99,25 @@ def run(self):
         config.intra_op_parallelism_threads=self.args.num_intra_threads
         config.inter_op_parallelism_threads=self.args.num_inter_threads
 
+        if self.args.onednn_graph:
+            import intel_extension_for_tensorflow as itex
+            auto_mixed_precision_options = itex.AutoMixedPrecisionOptions()
+            auto_mixed_precision_options.data_type = itex.BFLOAT16
+
+            auto_mixed_precision_options.allowlist_add= "Rsqrt,SquaredDifference,Mean"
+            auto_mixed_precision_options.denylist_remove = "Mean"
+            auto_mixed_precision_options.denylist_add = "QuantizeV2,Dequantize"
+
+            graph_options = itex.GraphOptions(auto_mixed_precision_options=auto_mixed_precision_options)
+            graph_options.auto_mixed_precision = itex.ON
+
+            config = itex.ConfigProto(graph_options=graph_options)
+            try:
+                itex.set_backend("cpu", config)
+            except TypeError:
+                itex.set_config(config)
+            config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
+
         with tf.compat.v1.Session(graph=graph, config=config) as sess:
             print("Started warmup for {} steps...".format(warmup_steps))
             for j in range(warmup_steps):
diff --git a/models/image_segmentation/tensorflow/maskrcnn/training/gpu/EnableBF16.patch b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/EnableBF16.patch
new file mode 100644
index 000000000..53d313e62
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/EnableBF16.patch
@@ -0,0 +1,395 @@
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/main.py b/TensorFlow2/Segmentation/MaskRCNN/main.py
+index 53b59d6e..6902a0ba 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/main.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/main.py
+@@ -57,6 +57,20 @@ def main():
+     # setup dataset
+     dataset = Dataset(params)
+ 
++    try:
++      import horovod.tensorflow.keras as hvd
++      import tensorflow as tf
++      hvd.init()
++      params.mpi_num = hvd.size()
++      params.model_dir = params.model_dir if hvd.rank() == 0 else \
++          os.path.join(params.model_dir, str(hvd.rank()))
++      xpus = tf.config.experimental.list_physical_devices('XPU')
++      tf.config.experimental.set_visible_devices(xpus[hvd.local_rank()], 'XPU')
++      logging.info("MPI horovod is enabled, this is running in MPI mode! local_rank = {}".format(hvd.local_rank()))
++    except ImportError:
++      params.mpi_num = 0
++      logging.info("No MPI horovod support, this is running in no-MPI mode!")
++
+     if params.mode == 'train':
+         run_training(dataset, params)
+     if params.mode == 'eval':
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
+index 63368c09..ad04efa4 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
+@@ -53,6 +53,14 @@ RUNTIME_GROUP.add_argument(
+     ]
+ )
+ 
++LOGGING_GROUP.add_argument(
++    '--mpi_num',
++    type=int,
++    default=0,
++    metavar='N',
++    help='Number of mpi ranks'
++)
++
+ RUNTIME_GROUP.add_argument(
+     '--data_dir',
+     type=str,
+@@ -72,7 +80,7 @@ RUNTIME_GROUP.add_argument(
+ RUNTIME_GROUP.add_argument(
+     '--backbone_checkpoint',
+     type=str,
+-    default='/weights/rn50_tf_amp_ckpt_v20.06.0/nvidia_rn50_tf_amp',
++    default=None,
+     metavar='FILE',
+     help='Pretrained checkpoint for resnet'
+ )
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
+index 29d2373a..4bd0f2b6 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
+@@ -48,6 +48,32 @@ class Dataset:
+ 
+     def train_fn(self, batch_size):
+         """ Input function for training. """
++        # Features
++        if self._params.use_synthetic_data:
++            self._logger.info("Using synthetic data")
++            name_to_features = {
++                "source_ids": tf.zeros((batch_size, ), dtype=tf.float32),
++                "images": tf.zeros((batch_size, 1024, 1024, 3), dtype=tf.float32),
++                "image_info": tf.zeros((batch_size, 5,) , dtype=tf.float32),
++                "cropped_gt_masks": tf.zeros((batch_size, 100, 116, 116), dtype=tf.float32),
++                "gt_boxes": tf.zeros((batch_size, 100, 4) , dtype=tf.float32),
++                "gt_classes": tf.zeros((batch_size, 100, 1) , dtype=tf.float32),
++                "score_targets_2": tf.zeros((batch_size, 256, 256, 3), dtype=tf.float32),
++                "box_targets_2": tf.zeros((batch_size, 256, 256, 12), dtype=tf.float32),
++                "score_targets_3": tf.zeros((batch_size, 128, 128, 3), dtype=tf.float32),
++                "box_targets_3": tf.zeros((batch_size, 128, 128, 12), dtype=tf.float32),
++                "score_targets_4": tf.zeros((batch_size, 64, 64, 3), dtype=tf.float32),
++                "box_targets_4": tf.zeros((batch_size, 64, 64, 12), dtype=tf.float32),
++                "score_targets_5": tf.zeros((batch_size, 32, 32, 3), dtype=tf.float32),
++                "box_targets_5": tf.zeros((batch_size, 32, 32, 12), dtype=tf.float32),
++                "score_targets_6": tf.zeros((batch_size, 16, 16, 3), dtype=tf.float32),
++                "box_targets_6": tf.zeros((batch_size, 16, 16, 12), dtype=tf.float32),
++            }
++
++            return tf.data.Dataset.from_tensors((name_to_features, {})).repeat()
++
++        if self._params.mpi_num:
++            batch_size = batch_size * self._params.mpi_num
+         data = tf.data.TFRecordDataset(self._train_files)
+ 
+         data = data.cache()
+@@ -73,7 +99,9 @@ class Dataset:
+ 
+         data = data.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
+         data = data.with_options(self._data_options)
+-
++        if self._params.mpi_num:
++            import horovod.tensorflow.keras as hvd
++            data = data.shard(hvd.size(), hvd.rank())
+         return data
+ 
+     def eval_fn(self, batch_size):
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
+index 96da0e36..3c071b10 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
+@@ -99,7 +99,7 @@ def dataset_parser(value, mode, params, use_instance_mask, seed=None, regenerate
+ 
+     example_decoder = create_example_decoder()
+ 
+-    with tf.xla.experimental.jit_scope(compile_ops=True):
++    with tf.xla.experimental.jit_scope(compile_ops=False):
+ 
+         with tf.name_scope('parser'):
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
+index ba922500..e6455b3f 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
+@@ -240,7 +240,7 @@ class AnchorLabeler:
+             tf.constant(0, dtype=tf.int32, shape=match_results.shape))
+         ignore_labels = tf.fill(match_results.shape, -1)
+ 
+-        return (ignore_labels + positive_labels + negative_labels,
++        return (tf.cast(ignore_labels + positive_labels + negative_labels, tf.float32),
+                 positive_labels, negative_labels)
+ 
+     def label_anchors(self, gt_boxes, gt_labels):
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
+index f4e22985..b681d4c0 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
+@@ -28,8 +28,8 @@ class MaskRCNNLoss(tf.keras.layers.Layer):
+     so that it doesn't expand `mask_targets`.
+     """
+ 
+-    def __init__(self):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, name="MaskRCNNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+ 
+     def call(self, inputs, **kwargs):
+         """
+@@ -82,8 +82,8 @@ class FastRCNNLoss(tf.keras.layers.Layer):
+     Reference: https://github.com/facebookresearch/Detectron/blob/master/detectron/modeling/fast_rcnn_heads.py
+     """
+ 
+-    def __init__(self, num_classes):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, num_classes, name="FastRCNNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+         self._num_classes = num_classes
+ 
+     def call(self, inputs, **kwargs):
+@@ -155,8 +155,8 @@ class RPNLoss(tf.keras.layers.Layer):
+     Computes total RPN detection loss including box and score from all levels.
+     """
+ 
+-    def __init__(self, batch_size, rpn_batch_size_per_im, min_level, max_level):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, batch_size, rpn_batch_size_per_im, min_level, max_level, name="RPNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+         self._batch_size = batch_size
+         self._rpn_batch_size_per_im = rpn_batch_size_per_im
+         self._min_level = min_level
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
+index 1cc1e3fd..21b9f9a7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
+@@ -251,21 +251,21 @@ class MaskRCNN(tf.keras.Model):
+         mask_rcnn_loss = self.mask_rcnn_loss(model_outputs)
+         mask_rcnn_loss *= self._params.mrcnn_weight_loss_mask
+         self.add_loss(mask_rcnn_loss)
+-        self.add_metric(mask_rcnn_loss, name='mask_rcnn_loss')
++        # self.add_metric(mask_rcnn_loss, name='mask_rcnn_loss')
+ 
+         fast_rcnn_class_loss, fast_rcnn_box_loss = self.fast_rcnn_loss(model_outputs)
+         fast_rcnn_box_loss *= self._params.fast_rcnn_box_loss_weight
+         self.add_loss(fast_rcnn_box_loss)
+-        self.add_metric(fast_rcnn_box_loss, name='fast_rcnn_box_loss')
++        # self.add_metric(fast_rcnn_box_loss, name='fast_rcnn_box_loss')
+         self.add_loss(fast_rcnn_class_loss)
+-        self.add_metric(fast_rcnn_class_loss, name='fast_rcnn_class_loss')
++        # self.add_metric(fast_rcnn_class_loss, name='fast_rcnn_class_loss')
+ 
+         rpn_score_loss, rpn_box_loss = self.rpn_loss(model_outputs)
+         rpn_box_loss *= self._params.rpn_box_loss_weight
+         self.add_loss(rpn_box_loss)
+-        self.add_metric(rpn_box_loss, name='rpn_box_loss')
++        # self.add_metric(rpn_box_loss, name='rpn_box_loss')
+         self.add_loss(rpn_score_loss)
+-        self.add_metric(rpn_score_loss, name='rpn_score_loss')
++        # self.add_metric(rpn_score_loss, name='rpn_score_loss')
+ 
+         l2_regularization_loss = tf.add_n([
+             tf.nn.l2_loss(tf.cast(v, dtype=tf.float32))
+@@ -274,7 +274,7 @@ class MaskRCNN(tf.keras.Model):
+         ])
+         l2_regularization_loss *= self._params.l2_weight_decay
+         self.add_loss(l2_regularization_loss)
+-        self.add_metric(l2_regularization_loss, name='l2_regularization_loss')
++        # self.add_metric(l2_regularization_loss, name='l2_regularization_loss')
+ 
+     def get_config(self):
+         pass
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+index 8670753e..deaf434a 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+@@ -96,16 +96,16 @@ def _propose_rois_gpu(scores,
+         pre_nms_boxes = tf.cast(pre_nms_boxes, dtype=tf.float32)
+         pre_nms_scores = tf.cast(pre_nms_scores, dtype=tf.float32)
+ 
+-        with tf.device('CPU:0'):
+-            boxes, scores, _, _ = tf.image.combined_non_max_suppression(
+-                pre_nms_boxes,
+-                pre_nms_scores,
+-                max_output_size_per_class=topk_limit,
+-                max_total_size=post_nms_topk_limit,
+-                iou_threshold=rpn_nms_threshold,
+-                score_threshold=0.0,
+-                pad_per_class=False
+-            )
++        # with tf.device('CPU:0'):
++        boxes, scores, _, _ = tf.image.combined_non_max_suppression(
++            pre_nms_boxes,
++            pre_nms_scores,
++            max_output_size_per_class=topk_limit,
++            max_total_size=post_nms_topk_limit,
++            iou_threshold=rpn_nms_threshold,
++            score_threshold=0.0,
++            pad_per_class=False
++        )
+ 
+         boxes = box_utils.to_absolute_coordinates(boxes, height, width)
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+index 4e7b56c9..20ae757c 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+@@ -127,12 +127,12 @@ class DLLoggerPerfCallback(KerasCallback):
+         self._dllogger.log(
+             step=step,
+             data={
++                f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time()),
+                 f'{mode}_throughput': self._calculate_throughput(deltas, self._batch_sizes[mode]),
+-                f'{mode}_latency': self._calculate_latency(deltas),
+-                f'{mode}_latency_90': self._calculate_latency_confidence(deltas, 90.0),
+-                f'{mode}_latency_95': self._calculate_latency_confidence(deltas, 95.0),
+-                f'{mode}_latency_99': self._calculate_latency_confidence(deltas, 99.0),
+-                f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time())
++                # f'{mode}_latency': self._calculate_latency(deltas),
++                # f'{mode}_latency_90': self._calculate_latency_confidence(deltas, 90.0),
++                # f'{mode}_latency_95': self._calculate_latency_confidence(deltas, 95.0),
++                # f'{mode}_latency_99': self._calculate_latency_confidence(deltas, 99.0),
+             }
+         )
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
+index ee96214e..a3566a1a 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
+@@ -1,6 +1,6 @@
+ import tensorflow as tf
+ 
+-
++@tf.keras.utils.register_keras_serializable('PiecewiseConstantWithWarmupSchedule')
+ class PiecewiseConstantWithWarmupSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
+     """
+     Schedule that starts with learning rate at `init_value` and monotonically increases
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+index d7b001b3..f0854fd7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+@@ -14,35 +14,39 @@ from mrcnn_tf2.runtime.weights_mapping import WEIGHTS_MAPPING
+ def run_training(dataset, params):
+     setup(params)
+ 
+-    strategy = tf.distribute.MirroredStrategy()
+-    params.replicas = strategy.num_replicas_in_sync
++    params.replicas = 1
+     params.global_train_batch_size = params.train_batch_size * params.replicas
+-    logging.info(f'Distributed Strategy is activated for {params.replicas} device(s)')
+-
+-    with strategy.scope():
+-
+-        learning_rate = PiecewiseConstantWithWarmupSchedule(
+-            init_value=params.init_learning_rate,
+-            # scale boundaries from epochs to steps
+-            boundaries=[
+-                int(b * dataset.train_size / params.global_train_batch_size)
+-                for b in params.learning_rate_boundaries
+-            ],
+-            values=params.learning_rate_values,
+-            # scale only by local BS as distributed strategy later scales it by number of replicas
+-            scale=params.train_batch_size
+-        )
+ 
+-        optimizer = tf.keras.optimizers.SGD(
+-            learning_rate=learning_rate,
+-            momentum=params.momentum
+-        )
++    learning_rate_init_value = params.init_learning_rate
++    if params.mpi_num > 1:
++        learning_rate_init_value = learning_rate_init_value * params.mpi_num
++    learning_rate = PiecewiseConstantWithWarmupSchedule(
++        init_value=learning_rate_init_value,
++        # scale boundaries from epochs to steps
++        boundaries=[
++            int(b * dataset.train_size / params.global_train_batch_size)
++            for b in params.learning_rate_boundaries
++        ],
++        values=params.learning_rate_values,
++        # scale only by local BS as distributed strategy later scales it by number of replicas
++        scale=params.train_batch_size
++    )
+ 
+-        mask_rcnn_model = create_model(params)
++    optimizer = tf.keras.optimizers.legacy.SGD(
++        learning_rate=learning_rate,
++        momentum=params.momentum
++    )
++    if params.mpi_num > 1:
++        import horovod.tensorflow.keras as hvd
++        logging.info("MPI horovod is enabled, using hvd.DistributedOptimizer")
++        optimizer = hvd.DistributedOptimizer(optimizer, groups=1)
+ 
+-        mask_rcnn_model.compile(
+-            optimizer=optimizer
+-        )
++    mask_rcnn_model = create_model(params)
++
++    mask_rcnn_model.compile(
++        optimizer=optimizer,
++        experimental_run_tf_function=False
++    )
+ 
+     # distributed strategy splits data between instances so we need global BS
+     train_data = dataset.train_fn(batch_size=params.global_train_batch_size)
+@@ -112,8 +116,9 @@ def setup(params):
+         logging.info('XLA is activated')
+ 
+     if params.amp:
+-        policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16", loss_scale="dynamic")
+-        tf.keras.mixed_precision.experimental.set_policy(policy)
++        # change precision from mixed_float16 to mixed_bfloat16
++        policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++        tf.keras.mixed_precision.set_global_policy(policy)
+         logging.info('AMP is activated')
+ 
+ 
+@@ -164,13 +169,26 @@ def create_callbacks(params):
+             mapping=lambda name: WEIGHTS_MAPPING.get(name.replace(':0', ''), name)
+         )
+ 
+-    yield tf.keras.callbacks.ModelCheckpoint(
+-        filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
+-        verbose=1
+-    )
++    # if params.mpi_num > 1:
++    #     import horovod.tensorflow.keras as hvd
++    #     if hvd.rank() == 0:
++    #         yield tf.keras.callbacks.ModelCheckpoint(
++    #             filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
++    #             verbose=1
++    #         )
++    # else:
++    #     yield tf.keras.callbacks.ModelCheckpoint(
++    #         filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
++    #         verbose=1
++    #     )
+ 
+     if params.log_tensorboard:
+         yield tf.keras.callbacks.TensorBoard(
+-            log_dir=params.log_tensorboard,
+-            update_freq='batch'
++            log_dir=params.log_tensorboard, profile_batch=0,
++            update_freq=5
+         )
++
++    if params.mpi_num > 1:
++        import horovod.tensorflow.keras as hvd
++        logging.info("MPI horovod is enabled, hvd.callbacks.BroadcastGlobalVariablesCallback")
++        yield hvd.callbacks.BroadcastGlobalVariablesCallback(0)
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py b/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
+index 6b61827e..e46ff590 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
+@@ -62,7 +62,6 @@ if __name__ == '__main__':
+         f'python {main_path}'
+         f' train'
+         f' --data_dir "{flags.data_dir}"'
+-        f' --backbone_checkpoint "{checkpoint_path}"'
+         f' --train_batch_size {flags.batch_size}'
+     )
+     cmd_eval = (
diff --git a/models/image_segmentation/tensorflow/maskrcnn/training/gpu/README.md b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/README.md
new file mode 100644
index 000000000..98ba845bf
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/README.md
@@ -0,0 +1,65 @@
+# Mask RCNN training
+
+Mask RCNN Training BKC.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Weight** | **Optional Patch** |
+| :---: | :---: | :---: | :---: | :---: | :---: |
+|   training   |  Tensorflow   | [DeepLearningExamples/MaskRCNN](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN) |        master         | See Section [Prerequisites](#weight) | [EnableBF16.patch](#bf16patch) |
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU MAX or FLEX
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+
+# Dataset 
+Download & preprocess COCO 2017 dataset. 
+```
+export DATASET_DIR=/path/to/dataset/dir
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/dataset
+bash dataset/download_and_preprocess_coco.sh $DATASET_DIR
+```
+
+## Training
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models/image_segmentation/tensorflow/maskrcnn/training/gpu`
+3. Run `setup.sh` this will install all the required dependencies & create virtual environment `venv`.
+4. Activate virtual env: `. ./venv/bin/activate`
+5. Download weights
+    ```
+        pushd .
+        cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+        python scripts/download_weights.py --save_dir=./weights
+        popd 
+    ```
+5. Setup required environment paramaters
+
+    |   **Parameter**    | **export command**                                    |
+    | :---: | :--- |
+    |  **DATASET_DIR**   | `export DATASET_DIR=/the/path/to/dataset`             |
+    |   **OUTPUT_DIR**   | `export OUTPUT_DIR=/the/path/to/output_dir`           |
+    |   **BATCH_SIZE** (optional)   | `export BATCH_SIZE=4`           |
+    |   **PRECISION**   | `export PRECISION=bfloat16` (bfloat16 or fp32)           |
+    |   **EPOCHS** (optional)  | `export EPOCHS=1`           |
+    |   **STEPS_PER_EPOCH** (optional)  | `export STEPS_PER_EPOCH=20`           |
+6. Run `run_model.sh`
+
+## Output
+
+Output will typicall looks like:
+```
+2023-09-11 14:54:49,905 I dllogger        (1, 20) loss: 639.5632934570312
+2023-09-11 14:54:49,906 I dllogger        (1, 20) train_time: 23.89216899871826, train_throughput: 21.438093303125907
+2023-09-11 14:54:49,914 I dllogger        (1,) loss: 639.5632934570312
+2023-09-11 14:54:49,914 I dllogger        () loss: 639.5632934570312
+2023-09-11 14:54:49,915 I dllogger        () train_time: 23.90118169784546, train_throughput: 23.507529269636105
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 23.507529269636105
+   unit: images/sec
+```
diff --git a/models/image_segmentation/tensorflow/maskrcnn/training/gpu/requirements.txt b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/requirements.txt
new file mode 100644
index 000000000..d1202f8b8
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/requirements.txt
@@ -0,0 +1,7 @@
+tensorflow
+intel-extension-for-tensorflow[xpu]
+opencv-python-headless
+pybind11
+pycocotools
+numpy<1.24
+-e "git+https://github.com/NVIDIA/dllogger#egg=dllogger"
diff --git a/models/image_segmentation/tensorflow/maskrcnn/training/gpu/run_model.sh b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/run_model.sh
new file mode 100755
index 000000000..636394fed
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/run_model.sh
@@ -0,0 +1,77 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: EPL-2.0
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[DATASET_DIR]=${DATASET_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d ${DATASET_DIR} ];then
+  echo "DATASET_DIR is: "${DATASET_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+BATCH_SIZE=${BATCH_SIZE:-4}
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+EPOCHS=${EPOCHS:-1}
+STEPS_PER_EPOCH=${STEPS_PER_EPOCH:-20}
+
+if [ ${PRECISION} == "bfloat16" ];then
+  echo "PRECISION is bfloat16"
+  AMP="--amp"
+else
+  echo "PRECISION is "${PRECISION}
+  AMP=""
+fi
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: $BATCH_SIZE"
+echo " EPOCHS: $EPOCHS"
+echo " STEPS_PER_EPOCH: $STEPS_PER_EPOCH" 
+
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+
+python main.py train \
+--data_dir $DATASET_DIR \
+--model_dir=$OUTPUT_DIR \
+--train_batch_size $BATCH_SIZE \
+--seed=0 --use_synthetic_data \
+--epochs $EPOCHS --steps_per_epoch $STEPS_PER_EPOCH \
+--log_every=1 --log_warmup_steps=1 \
+$AMP
+
diff --git a/docker/dataset/Makefile b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/setup.sh
old mode 100644
new mode 100755
similarity index 52%
rename from docker/dataset/Makefile
rename to models/image_segmentation/tensorflow/maskrcnn/training/gpu/setup.sh
index 2d9727090..fffd2202b
--- a/docker/dataset/Makefile
+++ b/models/image_segmentation/tensorflow/maskrcnn/training/gpu/setup.sh
@@ -1,5 +1,4 @@
-#
-# -*- coding: utf-8 -*-
+#!/usr/bin/env bash
 #
 # Copyright (c) 2023 Intel Corporation
 #
@@ -16,12 +15,20 @@
 # limitations under the License.
 #
 
-#
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+python3 -m venv $PWD/venv
+. ./venv/bin/activate
+pip install -r requirements.txt
 
-TENSORFLOW_IMAGE=intel/intel-optimized-tensorflow
-TENSORFLOW_TAG=1.15.2
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+git checkout c481324031ecf0f70f8939516c02e16cac60446d
+git apply  ../../../../EnableBF16.patch
+cd -
 
-all:
-	@TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE} \
-	 TENSORFLOW_TAG=${TENSORFLOW_TAG} \
-	 docker compose -f dataset-docker-compose.yml up --build
diff --git a/models/language_modeling/pytorch/bert_large/inference/gpu/cmd.sh b/models/language_modeling/pytorch/bert_large/inference/gpu/cmd.sh
deleted file mode 100755
index 0b9aa2c68..000000000
--- a/models/language_modeling/pytorch/bert_large/inference/gpu/cmd.sh
+++ /dev/null
@@ -1,70 +0,0 @@
-#!/bin/bash
-#
-# Copyright (c) 2022-2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-NUMA_ARGS=""
-if command -v numactl >& /dev/null ; then
-if [ "x$MPI_LOCALRANKID" != "x" ] ; then
-  REAL_NUM_NUMA_NODES=`lscpu | grep "NUMA node(s):" | awk '{print $NF}'`
-  PPNUMA=$(( MPI_LOCALNRANKS / REAL_NUM_NUMA_NODES ))
-  if [ $PPNUMA -eq 0 ] ; then 
-    if [ "x$SINGLE_SOCKET_ONLY" == "x1" ] ; then 
-      NUMA_ARGS="numactl -m 0 "
-    fi
-  else
-    NUMARANK=$(( MPI_LOCALRANKID / PPNUMA ))
-    NUMA_ARGS="$NUMA_ARGS $GDB_ARGS "
-  fi
-  NUM_RANKS=$PMI_SIZE
-else
-  NUMA_ARGS="numactl -m 0 "
-  NUM_RANKS=1
-fi
-fi
-
-if [ "x$1" == "x-gdb" ] ; then
-GDB_ARGS="gdb --args "
-shift
-else
-GDB_ARGS=""
-fi
-
-# set dataset
-if test -z $dataset || ! test -d $dataset ; then
-  if test -d ./SQUAD1 ; then
-    dataset=./SQUAD1
-  else
-    echo "Unable to find SQUAD dataset path"
-    exit 1
-  fi
-fi
-
-$NUMA_ARGS $GDB_ARGS python -u run_squad.py \
-  --model_type bert \
-  --model_name_or_path bert-large-uncased-whole-word-masking \
-  --do_train \
-  --do_eval \
-  --do_lower_case \
-  --train_file $dataset/train-v1.1.json \
-  --predict_file $dataset/dev-v1.1.json \
-  --per_gpu_train_batch_size 24 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 2.0 \
-  --max_seq_length 384 \
-  --doc_stride 128 \
-  --output_dir /tmp/debug_squad/ \
-  $@
-
diff --git a/models/language_modeling/pytorch/bert_large/inference/gpu/cmd_infer.sh b/models/language_modeling/pytorch/bert_large/inference/gpu/cmd_infer.sh
index 0afbca71e..3ae259ae8 100755
--- a/models/language_modeling/pytorch/bert_large/inference/gpu/cmd_infer.sh
+++ b/models/language_modeling/pytorch/bert_large/inference/gpu/cmd_infer.sh
@@ -1,5 +1,4 @@
 #!/bin/bash
-#
 # Copyright (c) 2022-2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-#
 
 function Parser() {
     while [ $# -ne 0 ]; do
@@ -99,30 +97,6 @@ else
 fi
 fi
 
-# set dataset and model_path
-if test -z $dataset || ! test -d $dataset ; then
-  dataset=$DATASET_DIR
-fi
-
-# if [ "$MODEL" == "bert_base" ] ; then
-#   if test -d ./squad_base_finetuned_checkpoint ; then
-#     :
-#   else
-#     ./download_squad_base_fine_tuned_model.sh
-#   fi
-#   model_path=./squad_base_finetuned_checkpoint
-# elif [ "$MODEL" == "bert_large" ] ; then
-#   if test -d ./squad_large_finetuned_checkpoint ; then
-#     :
-#   else
-#     ./download_squad_large_fine_tuned_model.sh
-#   fi
-#   model_path=./squad_large_finetuned_checkpoint 
-# else
-#   echo "The modle (${MODEL}) does not exist."
-#   exit
-# fi
-
 $NUMA_RAGS $GDB_ARGS python -u run_squad.py \
   --model_type bert \
   --model_name_or_path ${BERT_WEIGHT} \
@@ -131,7 +105,7 @@ $NUMA_RAGS $GDB_ARGS python -u run_squad.py \
   --do_jit \
   --device_choice ${DEVICE} \
   --dtype ${DTYPE}    \
-  --predict_file $dataset/dev-v1.1.json \
+  --predict_file $DATASET_DIR/dev-v1.1.json \
   --per_gpu_eval_batch_size ${BATCH} \
   --max_seq_length 384 \
   --doc_stride 128 \
diff --git a/models/language_modeling/pytorch/bert_large/inference/gpu/download_squad_large_fine_tuned_model.sh b/models/language_modeling/pytorch/bert_large/inference/gpu/download_squad_large_fine_tuned_model.sh
deleted file mode 100755
index d60d97cd9..000000000
--- a/models/language_modeling/pytorch/bert_large/inference/gpu/download_squad_large_fine_tuned_model.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/bash
-#
-# Copyright (c) 2022-2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/tree/main
-#mkdir bert-large-uncased-whole-word-masking-finetuned-squad
-set -e
-mkdir squad_large_finetuned_checkpoint && cd squad_large_finetuned_checkpoint
-wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json
-wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/pytorch_model.bin
-wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer.json
-wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer_config.json
-wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/vocab.txt
-
diff --git a/models/language_modeling/pytorch/bert_large/inference/gpu/run_squad.py b/models/language_modeling/pytorch/bert_large/inference/gpu/run_squad.py
index d5ed19a07..1284893f1 100644
--- a/models/language_modeling/pytorch/bert_large/inference/gpu/run_squad.py
+++ b/models/language_modeling/pytorch/bert_large/inference/gpu/run_squad.py
@@ -144,9 +144,9 @@ def load_model(args, config = None, checkpoint = None):
         )
     return model
 
-def load_jit_model(model, inputs, dtype, device, jit_trace_path):
+def load_jit_model(model, inputs, dtype, device, jit_trace_path, use_jit_cache):
     jit_model = None
-    if (os.path.isfile(jit_trace_path)):
+    if (os.path.isfile(jit_trace_path)) and use_jit_cache:
         print("load trace model ...")
         jit_model = torch.jit.load(jit_trace_path)
         print("load trace model done")
@@ -390,6 +390,11 @@ def train(args, train_dataset, model, tokenizer):
                 record_shapes=record_shapes,
             ) as prof:
                 # Skip past any already trained steps if resuming training
+                try:
+                    import memory_check
+                    memory_check.display_mem("xpu:0")
+                except:
+                    pass
                 if steps_trained_in_current_epoch > 0:
                     steps_trained_in_current_epoch -= 1
                     continue
@@ -652,7 +657,7 @@ def evaluate(args, model, tokenizer, prefix=""):
         if iter_num == 0:
            model = torch.xpu.optimize(model=model, dtype=MAP_TORCH_DTYPE[args.dtype], level="O1", weights_prepack = False)
            if args.do_jit:
-               jit_model = load_jit_model(model, inputs, MAP_TORCH_DTYPE[args.dtype], args.device, jit_trace_path)
+               jit_model = load_jit_model(model, inputs, MAP_TORCH_DTYPE[args.dtype], args.device, jit_trace_path, args.jit_cache)
         with torch.no_grad():
             outputs = None
             batch_start = None
@@ -1008,6 +1013,12 @@ def main():
     parser.add_argument(
         "--do_jit", action="store_true", help="Whether to run eval with jit on the dev set."
     )
+    parser.add_argument(
+        "--jit_cache",
+        action="store_true",
+        default=True,
+        help="Whether to use jit model cache",
+    )
     parser.add_argument(
         "--evaluate_during_training",
         action="store_true",
diff --git a/models/language_modeling/pytorch/fast_bert/pretrain_mlperf/run_pretrain_mlperf.py b/models/language_modeling/pytorch/fast_bert/pretrain_mlperf/run_pretrain_mlperf.py
index 040e3eeb0..98e94aa03 100644
--- a/models/language_modeling/pytorch/fast_bert/pretrain_mlperf/run_pretrain_mlperf.py
+++ b/models/language_modeling/pytorch/fast_bert/pretrain_mlperf/run_pretrain_mlperf.py
@@ -71,7 +71,7 @@
 import mlperf_logger
 from lamb import Lamb
 import intel_extension_for_pytorch as ipex
-import intel_extension_for_pytorch.tpp as tpp
+from intel_extension_for_pytorch import tpp as tpp
 
 
 ref_time = 0
@@ -1424,8 +1424,11 @@ def main():
                 # dataset_future = pool.submit(create_pretraining_dataset, data_file, args.max_predictions_per_seq, shared_file_list, args, worker_init_fn=worker_init)
                 pass
             t0 = get_time()
+            bench_time = 0
+            completed_steps = 0
             for step, batch in enumerate(train_dataloader):
                 training_steps += 1
+                t_beg=time.time()
                 t1 = get_time()
                 (
                     input_ids,
@@ -1498,6 +1501,11 @@ def main():
                     optimizer.acc_and_zero_grad()
                 # mlperf_logger.barrier()
                 t5 = get_time()
+                t_end=time.time()
+                completed_steps += 1
+                if global_step >= 10:
+                    bench_time += (t_end - t_beg)
+
                 gloss, lm_acc, num_masked, seq_acc, seq_tot = calc_accuracy(
                     outputs, masked_lm_labels, next_sentence_labels, args
                 )
@@ -1564,10 +1572,12 @@ def print_summary(desc, t):
                         print_summary("SYS:", t_all_time[:, 7])
                         print_summary("ARS:", t_all_time[:, 8])
                         print_summary("ARE:", t_all_time[:, 9])
-                if args.benchmark_steps > 0 and global_step + 1 >= args.benchmark_steps:
+                if args.benchmark_steps > 0 and completed_steps + 1  >= args.benchmark_steps:
                     mlperf_logger.barrier()
                     if args.local_rank == 0:
                         print(f"Done Benchmarking {args.benchmark_steps} steps.")
+                        throughput = ( completed_steps - 10 ) *   total_batch_size / bench_time
+                        print("Throughput: {:.3f} sentence/s".format(throughput), flush=True)
                     sys.exit(0)
 
                 update_step = training_steps % args.gradient_accumulation_steps == 0
diff --git a/models/language_modeling/tensorflow/bert_large/inference/generic_ops.py b/models/language_modeling/tensorflow/bert_large/inference/generic_ops.py
index a8a4576a1..f815b83a3 100755
--- a/models/language_modeling/tensorflow/bert_large/inference/generic_ops.py
+++ b/models/language_modeling/tensorflow/bert_large/inference/generic_ops.py
@@ -33,8 +33,6 @@ def set_global_precision(dt):
   global _keras_policy
   if dt == tf.bfloat16:
     _keras_policy=tf.keras.mixed_precision.Policy("mixed_bfloat16")
-  elif dt == tf.float16:
-    _keras_policy=tf.keras.mixed_precision.Policy("mixed_float16")
 
   # Set basic API precision
   set_rprecision(dt)
@@ -64,7 +62,7 @@ def multiply(x,y):
     return tf.multiply(x,y)
 
 def mzip(x,y):
-    if x.dtype== tf.bfloat16 or x.dtype == tf.float16:
+    if x.dtype== tf.bfloat16:
       x = r_cast(x)
       y = r_cast(y)
     return zip(x,y)
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/export_classifier.py b/models/language_modeling/tensorflow/bert_large/inference/gpu/export_classifier.py
new file mode 100644
index 000000000..3a47d1ec5
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/export_classifier.py
@@ -0,0 +1,181 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+
+import tensorflow as tf
+from absl import app
+from absl import logging
+tf.compat.v1.disable_v2_behavior()
+
+tf.compat.v1.flags.DEFINE_bool("saved_model",
+                               False,
+                               "whether export saved model or not")
+FLAGS = tf.compat.v1.flags.FLAGS
+
+# We just import classifier here for `create_model` and some processors such as
+# MNLI or MRPC. Because of the flags defined in `run_classifier.py`, we need not
+# to define the flags again.
+from run_classifier import create_model_top
+from run_classifier import ColaProcessor
+from run_classifier import MnliProcessor
+from run_classifier import MrpcProcessor
+from run_classifier import XnliProcessor
+from modeling import BertConfig
+
+class ClassifierExporter:
+  def __init__(self,
+               output_dir: str,
+               task_name: str,
+               bert_config: str,
+               max_seq_length: int):
+
+    processors = {
+        "cola": ColaProcessor,
+        "mnli": MnliProcessor,
+        "mrpc": MrpcProcessor,
+        "xnli": XnliProcessor
+    }
+
+    task_name = task_name.lower()
+    if task_name not in processors:
+      raise ValueError("Task not found: %s" % (task_name))
+
+    processor = processors[task_name]()
+    label_list = processor.get_labels()
+    num_labels = len(label_list)
+
+    # create model for CPU/dGPU, not TPU
+    use_one_hot_embeddings = False
+
+    bert_config = BertConfig.from_json_file(bert_config)
+    bert_config.experimental_gelu = FLAGS.experimental_gelu
+    if FLAGS.precision:
+      bert_config.precision = FLAGS.precision
+
+    self.session = tf.compat.v1.Session()
+
+    placeholder = tf.compat.v1.placeholder
+    input_shape = [None, max_seq_length]
+    self.label_ids = placeholder(tf.int32, [None], name='label_ids')
+    self.input_ids = placeholder(tf.int32, input_shape, name='input_ids')
+    self.input_mask = placeholder(tf.int32, input_shape, name='input_mask')
+    self.segment_ids = placeholder(tf.int32, input_shape, name='segment_ids')
+
+    self.loss, self.per_example_loss, self.logits, self.probabilities = \
+      create_model_top(bert_config, False, # is training
+                       self.input_ids, self.input_mask, self.segment_ids,
+                       self.label_ids, num_labels, use_one_hot_embeddings,
+                       None) # frozen graph path
+
+    latest_model = tf.train.latest_checkpoint(FLAGS.output_dir)
+    saver = tf.compat.v1.train.Saver()
+    saver.restore(self.session, latest_model)
+
+    self.output_dir = output_dir
+    self.dest_dir = os.path.join(self.output_dir, "frozen")
+    if not os.path.exists(self.dest_dir):
+      os.mkdir(self.dest_dir)
+
+  def __enter__(self):
+    return self
+
+  def __exit__(self, exc_type, exc_value, exc_tb):
+    self.session.close()
+
+  def export(self, saved_model: bool):
+    if saved_model:
+      self.export_saved_model()
+
+    self.export_frozen_graph()
+
+  def export_saved_model(self,
+                         signature_def_name="eval",
+                         tag=tf.compat.v1.saved_model.tag_constants.SERVING):
+    build_tensor_info = tf.compat.v1.saved_model.build_tensor_info
+    signature_def_utils = tf.compat.v1.saved_model.signature_def_utils
+    inputs = {
+        'label_ids': build_tensor_info(self.label_ids),
+        'input_ids': build_tensor_info(self.input_ids),
+        'input_mask': build_tensor_info(self.input_mask),
+        'segment_ids': build_tensor_info(self.segment_ids)
+    }
+
+    outputs = {
+        "loss": build_tensor_info(self.loss),
+        "per_example_loss": build_tensor_info(self.per_example_loss),
+        "logits": build_tensor_info(self.logits),
+        "probabilities": build_tensor_info(self.probabilities)
+    }
+
+    signature = signature_def_utils.build_signature_def(inputs, outputs)
+    signature_def_map = {signature_def_name: signature}
+
+    builder = tf.compat.v1.saved_model.builder.SavedModelBuilder(self.dest_dir)
+    builder.add_meta_graph_and_variables(self.session, [tag], signature_def_map)
+    builder.save()
+
+  def export_frozen_graph(self, frozen_graph_name="frozen_graph.pb"):
+    # we should disable v2 behavior, at the same time, the bn norm has some op name difference
+    # should be handled. Otherwise, it will throw exception when do import graph def.
+    # https://www.bountysource.com/issues/36614355-unable-to-import-frozen-graph-with-batchnorm
+    graph_def = self.session.graph.as_graph_def()
+    for node in graph_def.node:
+      if node.op == 'RefEnter':
+        node.op = 'Enter'
+        for index in range(len(node.input)):
+          if 'moving_' in node.input[index]:
+            node.input[index] = node.input[index] + '/read'
+      if node.op == 'RefSwitch':
+        node.op = 'Switch'
+        for index in range(len(node.input)):
+          if 'moving_' in node.input[index]:
+            node.input[index] = node.input[index] + '/read'
+      elif node.op == 'AssignSub':
+        node.op = 'Sub'
+        if 'use_locking' in node.attr: del node.attr['use_locking']
+      elif node.op == 'AssignAdd':
+        node.op = 'Add'
+        if 'use_locking' in node.attr: del node.attr['use_locking']
+
+    outputs_name = ['loss/Mean', 'loss/Sum', 'loss/BiasAdd', 'loss/Softmax']
+    graph_def = tf.compat.v1.graph_util.convert_variables_to_constants(self.session,
+                                                                       graph_def,
+                                                                       outputs_name)
+
+    path = os.path.join(self.dest_dir, frozen_graph_name)
+    with tf.compat.v1.gfile.GFile(path, 'wb') as pb_file:
+      pb_file.write(graph_def.SerializeToString())
+
+def main(_):
+  tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
+
+  with ClassifierExporter(FLAGS.output_dir,
+                          FLAGS.task_name,
+                          FLAGS.bert_config_file,
+                          FLAGS.max_seq_length) as exporter:
+    exporter.export(FLAGS.saved_model)
+
+if __name__ == "__main__":
+  tf.compat.v1.flags.mark_flag_as_required("task_name")
+  tf.compat.v1.flags.mark_flag_as_required("bert_config_file")
+  tf.compat.v1.flags.mark_flag_as_required("output_dir")
+  tf.compat.v1.app.run()
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/generic_ops.py b/models/language_modeling/tensorflow/bert_large/inference/gpu/generic_ops.py
new file mode 100755
index 000000000..4de663cfb
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/generic_ops.py
@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""The main BERT model and related functions."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+import tensorflow as tf
+
+try:
+    from tensorflow.keras.mixed_precsion import experimental as mixed_precision
+except:
+    from tensorflow.keras import  mixed_precision
+
+_inprecision = tf.float32
+_rprecision = tf.float32
+_keras_policy = mixed_precision.Policy("float32")
+_use_optimized_softmax = True
+_use_experimental_gelu = False
+
+def set_global_precision(dt):
+  # Set Keras API precision
+  global _keras_policy
+  if dt == tf.bfloat16:
+    _keras_policy=mixed_precision.Policy("mixed_bfloat16")
+
+  # Set basic API precision
+  set_rprecision(dt)
+
+def set_rprecision(dt):
+  global _rprecision
+  _rprecision=dt
+
+def get_keras_policy():
+  return _keras_policy
+
+def set_global_flags(optimized_softmax, experimental_gelu):
+  global _use_optimized_softmax
+  global _use_experimental_gelu
+  _use_optimized_softmax = optimized_softmax
+  _use_experimental_gelu = experimental_gelu
+
+def i_cast(x) :
+     return tf.cast(x, _inprecision)
+
+def r_cast(x) :
+     return tf.cast(x, _rprecision)
+
+def multiply(x,y):
+    x = r_cast(x)
+    y = r_cast(y)
+    return tf.multiply(x,y)
+
+def mzip(x,y):
+    if x.dtype== tf.bfloat16:
+      x = r_cast(x)
+      y = r_cast(y)
+    return zip(x,y)
+
+def tanh(x):
+    x = i_cast(x)
+    rval = tf.tanh(x)
+    return r_cast(rval)
+
+def softmax(scores, axis=None):
+    if _use_optimized_softmax:
+      return tf.nn.softmax(scores, axis)
+    else:
+      scores = i_cast(scores)
+      rval = tf.nn.softmax(scores, axis)
+      return r_cast(rval)
+
+def layer_norm(inputs, begin_norm_axis, begin_params_axis, scope):
+    lnorm = tf.keras.layers.LayerNormalization(dtype=get_keras_policy())
+    return lnorm(inputs)
+
+"Moved from modeling.py"
+def gelu(x):
+  """Gaussian Error Linear Unit.
+
+  This is a smoother version of the RELU.
+  Original paper: https://arxiv.org/abs/1606.08415
+  Args:
+    x: float Tensor to perform activation.
+
+  Returns:
+    `x` with the GELU activation applied.
+  """
+  if _use_experimental_gelu:
+    return tf.nn.gelu(features=x, approximate=True)
+  else:
+    x = i_cast(x)
+    cdf = 0.5 * (1.0 + tf.tanh(
+        (np.sqrt(2 / np.pi) * (x + 0.044715 * tf.pow(x, 3)))))
+    rval = x * cdf
+    return r_cast(rval)
+
+def logTheLossHook(total_loss, n):
+    return tf.compat.v1.train.LoggingTensorHook({"\t Loss " : total_loss}, every_n_iter=n)
+
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/modeling.py b/models/language_modeling/tensorflow/bert_large/inference/gpu/modeling.py
new file mode 100644
index 000000000..226c5f331
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/modeling.py
@@ -0,0 +1,1050 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""The main BERT model and related functions."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import copy
+import json
+import math
+import re
+import numpy as np
+import six
+import tensorflow as tf
+import generic_ops as bf
+
+
+class BertConfig(object):
+  """Configuration for `BertModel`."""
+
+  def __init__(self,
+               vocab_size,
+               hidden_size=768,
+               num_hidden_layers=12,
+               num_attention_heads=12,
+               intermediate_size=3072,
+               hidden_act="gelu",
+               hidden_dropout_prob=0.1,
+               attention_probs_dropout_prob=0.1,
+               max_position_embeddings=512,
+               type_vocab_size=16,
+               initializer_range=0.02,
+               precision="fp32",
+               new_bf16_scope=True):
+    """Constructs BertConfig.
+
+    Args:
+      vocab_size: Vocabulary size of `inputs_ids` in `BertModel`.
+      hidden_size: Size of the encoder layers and the pooler layer.
+      num_hidden_layers: Number of hidden layers in the Transformer encoder.
+      num_attention_heads: Number of attention heads for each attention layer in
+        the Transformer encoder.
+      intermediate_size: The size of the "intermediate" (i.e., feed-forward)
+        layer in the Transformer encoder.
+      hidden_act: The non-linear activation function (function or string) in the
+        encoder and pooler.
+      hidden_dropout_prob: The dropout probability for all fully connected
+        layers in the embeddings, encoder, and pooler.
+      attention_probs_dropout_prob: The dropout ratio for the attention
+        probabilities.
+      max_position_embeddings: The maximum sequence length that this model might
+        ever be used with. Typically set this to something large just in case
+        (e.g., 512 or 1024 or 2048).
+      type_vocab_size: The vocabulary size of the `token_type_ids` passed into
+        `BertModel`.
+      initializer_range: The stdev of the truncated_normal_initializer for
+        initializing all weight matrices.
+      precision: To enable fp32 or bfloat16 based training
+    """
+    self.vocab_size = vocab_size
+    self.hidden_size = hidden_size
+    self.num_hidden_layers = num_hidden_layers
+    self.num_attention_heads = num_attention_heads
+    self.hidden_act = hidden_act
+    self.intermediate_size = intermediate_size
+    self.hidden_dropout_prob = hidden_dropout_prob
+    self.attention_probs_dropout_prob = attention_probs_dropout_prob
+    self.max_position_embeddings = max_position_embeddings
+    self.type_vocab_size = type_vocab_size
+    self.initializer_range = initializer_range
+    self.precision = precision
+    self.new_bf16_scope = new_bf16_scope 
+    self.experimental_gelu = False
+    self.optimized_softmax = False
+
+  @classmethod
+  def from_dict(cls, json_object):
+    """Constructs a `BertConfig` from a Python dictionary of parameters."""
+    config = BertConfig(vocab_size=None)
+    for (key, value) in six.iteritems(json_object):
+      config.__dict__[key] = value
+    return config
+
+  @classmethod
+  def from_json_file(cls, json_file):
+    """Constructs a `BertConfig` from a json file of parameters."""
+    with tf.io.gfile.GFile(json_file, "r") as reader:
+      text = reader.read()
+    return cls.from_dict(json.loads(text))
+
+  def to_dict(self):
+    """Serializes this instance to a Python dictionary."""
+    output = copy.deepcopy(self.__dict__)
+    return output
+
+  def to_json_string(self):
+    """Serializes this instance to a JSON string."""
+    return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+
+class BertModel(object):
+  """BERT model ("Bidirectional Encoder Representations from Transformers").
+
+  Example usage:
+
+  ```python
+  # Already been converted into WordPiece token ids
+  input_ids = tf.constant([[31, 51, 99], [15, 5, 0]])
+  input_mask = tf.constant([[1, 1, 1], [1, 1, 0]])
+  token_type_ids = tf.constant([[0, 0, 1], [0, 2, 0]])
+
+  config = modeling.BertConfig(vocab_size=32000, hidden_size=512,
+    num_hidden_layers=8, num_attention_heads=6, intermediate_size=1024)
+
+  model = modeling.BertModel(config=config, is_training=True,
+    input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_ids)
+
+  label_embeddings = tf.get_variable(...)
+  pooled_output = model.get_pooled_output()
+  logits = tf.matmul(pooled_output, label_embeddings)
+  ...
+  ```
+  """
+
+  def __init__(self,
+               config,
+               is_training,
+               input_ids,
+               input_mask=None,
+               token_type_ids=None,
+               use_one_hot_embeddings=False,
+               scope=None):
+    """Constructor for BertModel.
+
+    Args:
+      config: `BertConfig` instance.
+      is_training: bool. true for training model, false for eval model. Controls
+        whether dropout will be applied.
+      input_ids: int32 Tensor of shape [batch_size, seq_length].
+      input_mask: (optional) int32 Tensor of shape [batch_size, seq_length].
+      token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length].
+      use_one_hot_embeddings: (optional) bool. Whether to use one-hot word
+        embeddings or tf.embedding_lookup() for the word embeddings.
+      scope: (optional) variable scope. Defaults to "bert".
+
+    Raises:
+      ValueError: The config is invalid or one of the input tensor shapes
+        is invalid.
+    """
+    # Flags for BF16 CPU
+    self.bf16_scope = False
+    if config.precision == "bfloat16" :
+      bf.set_global_precision(tf.bfloat16)
+      if config.new_bf16_scope :
+        self.bf16_scope = True
+
+    bf.set_global_flags(optimized_softmax=config.optimized_softmax,
+                        experimental_gelu=config.experimental_gelu)
+
+    config = copy.deepcopy(config)
+    if not is_training:
+      config.hidden_dropout_prob = 0.0
+      config.attention_probs_dropout_prob = 0.0
+
+    input_shape = get_shape_list(input_ids, expected_rank=2)
+    batch_size = input_shape[0]
+    seq_length = input_shape[1]
+
+    if input_mask is None:
+      input_mask = tf.ones(shape=[batch_size, seq_length], dtype=tf.int32)
+
+    if token_type_ids is None:
+      token_type_ids = tf.zeros(shape=[batch_size, seq_length], dtype=tf.int32)
+
+    with tf.compat.v1.variable_scope(scope, default_name="bert"):
+      with tf.compat.v1.variable_scope("embeddings"):
+        # Perform embedding lookup on the word ids.
+        (self.embedding_output, self.embedding_table) = embedding_lookup(
+            input_ids=input_ids,
+            vocab_size=config.vocab_size,
+            embedding_size=config.hidden_size,
+            initializer_range=config.initializer_range,
+            word_embedding_name="word_embeddings",
+            use_one_hot_embeddings=use_one_hot_embeddings)
+
+        # Add positional embeddings and token type embeddings, then layer
+        # normalize and perform dropout.
+        self.embedding_output = embedding_postprocessor(
+            input_tensor=self.embedding_output,
+            use_token_type=True,
+            token_type_ids=token_type_ids,
+            token_type_vocab_size=config.type_vocab_size,
+            token_type_embedding_name="token_type_embeddings",
+            use_position_embeddings=True,
+            position_embedding_name="position_embeddings",
+            initializer_range=config.initializer_range,
+            max_position_embeddings=config.max_position_embeddings,
+            dropout_prob=config.hidden_dropout_prob)
+
+      with tf.compat.v1.variable_scope("encoder"):
+        # This converts a 2D mask of shape [batch_size, seq_length] to a 3D
+        # mask of shape [batch_size, seq_length, seq_length] which is used
+        # for the attention scores.
+        attention_mask = create_attention_mask_from_input_mask(
+            input_ids, input_mask)
+
+        # Run the stacked transformer.
+        # `sequence_output` shape = [batch_size, seq_length, hidden_size].
+        # Cast is used to cover bfloat16
+        input_tensor=bf.r_cast(self.embedding_output)
+        self.all_encoder_layers = transformer_model(
+            input_tensor=input_tensor,
+            attention_mask=attention_mask,
+            hidden_size=config.hidden_size,
+            num_hidden_layers=config.num_hidden_layers,
+            num_attention_heads=config.num_attention_heads,
+            intermediate_size=config.intermediate_size,
+            intermediate_act_fn=get_activation(config.hidden_act),
+            hidden_dropout_prob=config.hidden_dropout_prob,
+            attention_probs_dropout_prob=config.attention_probs_dropout_prob,
+            initializer_range=config.initializer_range,
+            do_return_all_layers=True)
+
+      self.sequence_output = self.all_encoder_layers[-1]
+      # The "pooler" converts the encoded sequence tensor of shape
+      # [batch_size, seq_length, hidden_size] to a tensor of shape
+      # [batch_size, hidden_size]. This is necessary for segment-level
+      # (or segment-pair-level) classification tasks where we need a fixed
+      # dimensional representation of the segment.
+      with tf.compat.v1.variable_scope("pooler"):
+        # We "pool" the model by simply taking the hidden state corresponding
+        # to the first token. We assume that this has been pre-trained
+        first_token_tensor = tf.squeeze(self.sequence_output[:, 0:1, :], axis=1)
+        self.pooled_output = tf.compat.v1.layers.dense(
+            first_token_tensor,
+            config.hidden_size,
+            activation=bf.tanh,
+            kernel_initializer=create_initializer(config.initializer_range))
+
+  def get_pooled_output(self):
+    """ 
+      In bfloat16 enabled execution, with only model covered in bfloat16 scope,
+      return the output in float32. Other cases return as is
+    """
+    if self.bf16_scope == True:
+      return  tf.cast(self.pooled_output, tf.float32)
+    else :
+      return self.pooled_output
+
+  def get_sequence_output(self):
+    """Gets final hidden layer of encoder.
+
+    Returns:
+      float Tensor of shape [batch_size, seq_length, hidden_size] corresponding
+      to the final hidden of the transformer encoder.
+
+      In bfloat16 enabled execution, with only model covered in bfloat16 scope,
+      return the output in float32. Other cases return as is
+    """
+    if self.bf16_scope == True:
+      return  tf.cast(self.sequence_output, tf.float32)
+    else :
+      return self.sequence_output
+
+  def get_all_encoder_layers(self):
+    return self.all_encoder_layers
+
+  def get_embedding_output(self):
+    """Gets output of the embedding lookup (i.e., input to the transformer).
+
+    Returns:
+      float Tensor of shape [batch_size, seq_length, hidden_size] corresponding
+      to the output of the embedding layer, after summing the word
+      embeddings with the positional embeddings and the token type embeddings,
+      then performing layer normalization. This is the input to the transformer.
+    """
+    if self.bf16_scope == True :
+      return (self.embedding_output) 
+    else :
+      return bf.r_cast(self.embedding_output)
+
+  def get_embedding_table(self):
+    if self.bf16_scope == True :
+      return (self.embedding_table) 
+    else :
+      return bf.r_cast(self.embedding_table)
+
+
+def get_activation(activation_string):
+  """Maps a string to a Python function, e.g., "relu" => `tf.nn.relu`.
+
+  Args:
+    activation_string: String name of the activation function.
+
+  Returns:
+    A Python function corresponding to the activation function. If
+    `activation_string` is None, empty, or "linear", this will return None.
+    If `activation_string` is not a string, it will return `activation_string`.
+
+  Raises:
+    ValueError: The `activation_string` does not correspond to a known
+      activation.
+  """
+
+  # We assume that anything that"s not a string is already an activation
+  # function, so we just return it.
+  if not isinstance(activation_string, six.string_types):
+    return activation_string
+
+  if not activation_string:
+    return None
+
+  act = activation_string.lower()
+  if act == "linear":
+    return None
+  elif act == "relu":
+    return tf.nn.relu
+  elif act == "gelu":
+    return bf.gelu
+  elif act == "tanh":
+    return bf.tanh
+  else:
+    raise ValueError("Unsupported activation: %s" % act)
+
+
+def get_remaps(task):
+  regex1 = re.compile(r"layer_normalization[_0-9]*")
+  regex2 = None 
+  if task=="SQuAD" :
+    regex2 = re.compile(r"squad")
+  elif task=="Classifier":
+    regex2 = re.compile(r"classifier")
+  else :
+    regex2 = None 
+
+  return regex1, regex2
+
+def apply_remaps(name, map1, map2):
+  if map1!=None:
+    name = map1.sub("LayerNorm", name)
+  if map2!=None :
+    name = map2.sub("seq_relationship", name)
+  return name 
+
+def check_model_validity(tvars, assignment_map) :
+  # Check if all model vars have a mapping in checkpoint
+  missing_var=False
+  missed_vars=[]
+  for var in tvars:
+     if var not in assignment_map.values():
+       missed_vars.append(var)
+       missing_var=True
+  if missing_var :
+     for var in missed_vars:
+       tf.compat.v1.logging.info("Model Variable not in checkpoint", var)
+     raise ValueError("Error: Missing model variables in checkpoint!!")
+
+def get_assignment_map_from_checkpoint(tvars, init_checkpoint, task="Pretraining"):
+  """Compute the union of the current variables and checkpoint variables."""
+  assignment_map = {}
+  initialized_variable_names = {}
+
+  map1, map2 = get_remaps(task)
+
+  name_to_variable = collections.OrderedDict()
+  for var in tvars:
+    name = var.name
+    m = re.match("^(.*):\\d+$", name)
+    if m is not None:
+      name = m.group(1)
+    name_to_variable[name] = var
+    name = apply_remaps(name, map1, map2)
+    name_to_variable[name] = var
+
+  init_vars = tf.train.list_variables(init_checkpoint)
+
+  assignment_map = collections.OrderedDict()
+  for x in init_vars:
+    (name, var) = (x[0], x[1])
+    if name not in name_to_variable:
+      continue
+    assignment_map[name] = name_to_variable[name]
+    ivar = name_to_variable[name]
+    initialized_variable_names[ivar.name] = 1
+    initialized_variable_names[ivar.name + ":0"] = 1
+
+  # Check if all model vars are loaded from Checkpoint
+  check_model_validity(tvars, assignment_map)
+  #for name, var in assignment_map.items():
+  #  print(name, "--->", var)
+
+  return (assignment_map, initialized_variable_names)
+
+def dropout(input_tensor, dropout_prob):
+  """Perform dropout.
+
+  Args:
+    input_tensor: float Tensor.
+    dropout_prob: Python float. The probability of dropping out a value (NOT of
+      *keeping* a dimension as in `tf.nn.dropout`).
+
+  Returns:
+    A version of `input_tensor` with dropout applied.
+  """
+  if dropout_prob is None or dropout_prob == 0.0:
+    return input_tensor
+
+  output = tf.nn.dropout(input_tensor, 1 - (1.0 - dropout_prob))
+  return output
+
+
+def layer_norm(input_tensor, name=None):
+  """Run layer normalization on the last dimension of the tensor."""
+  return bf.layer_norm(
+      inputs=input_tensor, begin_norm_axis=-1, begin_params_axis=-1, scope=name)
+
+
+def layer_norm_and_dropout(input_tensor, dropout_prob, name=None):
+  """Runs layer normalization followed by dropout."""
+  output_tensor = layer_norm(input_tensor, name)
+  output_tensor = dropout(output_tensor, dropout_prob)
+  return output_tensor
+
+
+def create_initializer(initializer_range=0.02):
+  """Creates a `truncated_normal_initializer` with the given range."""
+  return tf.compat.v1.truncated_normal_initializer(stddev=initializer_range)
+
+
+def embedding_lookup(input_ids,
+                     vocab_size,
+                     embedding_size=128,
+                     initializer_range=0.02,
+                     word_embedding_name="word_embeddings",
+                     use_one_hot_embeddings=False):
+  """Looks up words embeddings for id tensor.
+
+  Args:
+    input_ids: int32 Tensor of shape [batch_size, seq_length] containing word
+      ids.
+    vocab_size: int. Size of the embedding vocabulary.
+    embedding_size: int. Width of the word embeddings.
+    initializer_range: float. Embedding initialization range.
+    word_embedding_name: string. Name of the embedding table.
+    use_one_hot_embeddings: bool. If True, use one-hot method for word
+      embeddings. If False, use `tf.gather()`.
+
+  Returns:
+    float Tensor of shape [batch_size, seq_length, embedding_size].
+  """
+  # This function assumes that the input is of shape [batch_size, seq_length,
+  # num_inputs].
+  #
+  # If the input is a 2D tensor of shape [batch_size, seq_length], we
+  # reshape to [batch_size, seq_length, 1].
+  if input_ids.shape.ndims == 2:
+    input_ids = tf.expand_dims(input_ids, axis=[-1])
+
+  embedding_table = tf.compat.v1.get_variable(
+      name=word_embedding_name,
+      shape=[vocab_size, embedding_size],
+      initializer=create_initializer(initializer_range))
+
+  flat_input_ids = tf.reshape(input_ids, [-1])
+  if use_one_hot_embeddings:
+    one_hot_input_ids = tf.one_hot(flat_input_ids, depth=vocab_size)
+    output = tf.matmul(one_hot_input_ids, embedding_table)
+  else:
+    output = tf.gather(embedding_table, flat_input_ids)
+
+  input_shape = get_shape_list(input_ids)
+
+  output = tf.reshape(output,
+                      input_shape[0:-1] + [input_shape[-1] * embedding_size])
+  return (output, embedding_table)
+
+
+def embedding_postprocessor(input_tensor,
+                            use_token_type=False,
+                            token_type_ids=None,
+                            token_type_vocab_size=16,
+                            token_type_embedding_name="token_type_embeddings",
+                            use_position_embeddings=True,
+                            position_embedding_name="position_embeddings",
+                            initializer_range=0.02,
+                            max_position_embeddings=512,
+                            dropout_prob=0.1):
+  """Performs various post-processing on a word embedding tensor.
+
+  Args:
+    input_tensor: float Tensor of shape [batch_size, seq_length,
+      embedding_size].
+    use_token_type: bool. Whether to add embeddings for `token_type_ids`.
+    token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length].
+      Must be specified if `use_token_type` is True.
+    token_type_vocab_size: int. The vocabulary size of `token_type_ids`.
+    token_type_embedding_name: string. The name of the embedding table variable
+      for token type ids.
+    use_position_embeddings: bool. Whether to add position embeddings for the
+      position of each token in the sequence.
+    position_embedding_name: string. The name of the embedding table variable
+      for positional embeddings.
+    initializer_range: float. Range of the weight initialization.
+    max_position_embeddings: int. Maximum sequence length that might ever be
+      used with this model. This can be longer than the sequence length of
+      input_tensor, but cannot be shorter.
+    dropout_prob: float. Dropout probability applied to the final output tensor.
+
+  Returns:
+    float tensor with same shape as `input_tensor`.
+
+  Raises:
+    ValueError: One of the tensor shapes or input values is invalid.
+  """
+  input_shape = get_shape_list(input_tensor, expected_rank=3)
+  batch_size = input_shape[0]
+  seq_length = input_shape[1]
+  width = input_shape[2]
+
+  output = input_tensor
+
+  if use_token_type:
+    if token_type_ids is None:
+      raise ValueError("`token_type_ids` must be specified if"
+                       "`use_token_type` is True.")
+    token_type_table = tf.compat.v1.get_variable(
+        name=token_type_embedding_name,
+        shape=[token_type_vocab_size, width],
+        initializer=create_initializer(initializer_range))
+    # This vocab will be small so we always do one-hot here, since it is always
+    # faster for a small vocabulary.
+    flat_token_type_ids = tf.reshape(token_type_ids, [-1])
+    one_hot_ids = tf.one_hot(flat_token_type_ids, depth=token_type_vocab_size)
+    token_type_embeddings = tf.matmul(one_hot_ids, token_type_table)
+    token_type_embeddings = tf.reshape(token_type_embeddings,
+                                       [batch_size, seq_length, width])
+    output += token_type_embeddings
+
+  if use_position_embeddings:
+    assert_op = tf.compat.v1.assert_less_equal(seq_length, max_position_embeddings)
+    with tf.control_dependencies([assert_op]):
+      full_position_embeddings = tf.compat.v1.get_variable(
+          name=position_embedding_name,
+          shape=[max_position_embeddings, width],
+          initializer=create_initializer(initializer_range))
+      # Since the position embedding table is a learned variable, we create it
+      # using a (long) sequence length `max_position_embeddings`. The actual
+      # sequence length might be shorter than this, for faster training of
+      # tasks that do not have long sequences.
+      #
+      # So `full_position_embeddings` is effectively an embedding table
+      # for position [0, 1, 2, ..., max_position_embeddings-1], and the current
+      # sequence has positions [0, 1, 2, ... seq_length-1], so we can just
+      # perform a slice.
+      position_embeddings = tf.slice(full_position_embeddings, [0, 0],
+                                     [seq_length, -1])
+      num_dims = len(output.shape.as_list())
+
+      # Only the last two dimensions are relevant (`seq_length` and `width`), so
+      # we broadcast among the first dimensions, which is typically just
+      # the batch size.
+      position_broadcast_shape = []
+      for _ in range(num_dims - 2):
+        position_broadcast_shape.append(1)
+      position_broadcast_shape.extend([seq_length, width])
+      position_embeddings = tf.reshape(position_embeddings,
+                                       position_broadcast_shape)
+      output += position_embeddings
+
+  output = layer_norm_and_dropout(output, dropout_prob)
+  return output
+
+
+def create_attention_mask_from_input_mask(from_tensor, to_mask):
+  """Create 3D attention mask from a 2D tensor mask.
+
+  Args:
+    from_tensor: 2D or 3D Tensor of shape [batch_size, from_seq_length, ...].
+    to_mask: int32 Tensor of shape [batch_size, to_seq_length].
+
+  Returns:
+    float Tensor of shape [batch_size, from_seq_length, to_seq_length].
+  """
+  from_shape = get_shape_list(from_tensor, expected_rank=[2, 3])
+  batch_size = from_shape[0]
+  from_seq_length = from_shape[1]
+
+  to_shape = get_shape_list(to_mask, expected_rank=2)
+  to_seq_length = to_shape[1]
+
+  to_mask = tf.cast(
+      tf.reshape(to_mask, [batch_size, 1, to_seq_length]), tf.float32)
+
+  # We don't assume that `from_tensor` is a mask (although it could be). We
+  # don't actually care if we attend *from* padding tokens (only *to* padding)
+  # tokens so we create a tensor of all ones.
+  #
+  # `broadcast_ones` = [batch_size, from_seq_length, 1]
+  broadcast_ones = tf.ones(
+      shape=[batch_size, from_seq_length, 1], dtype=tf.float32)
+
+  # Here we broadcast along two dimensions to create the mask.
+  mask = broadcast_ones * to_mask
+
+  return mask
+
+
+def attention_layer(from_tensor,
+                    to_tensor,
+                    attention_mask=None,
+                    num_attention_heads=1,
+                    size_per_head=512,
+                    query_act=None,
+                    key_act=None,
+                    value_act=None,
+                    attention_probs_dropout_prob=0.0,
+                    initializer_range=0.02,
+                    do_return_2d_tensor=False,
+                    batch_size=None,
+                    from_seq_length=None,
+                    to_seq_length=None):
+  """Performs multi-headed attention from `from_tensor` to `to_tensor`.
+
+  This is an implementation of multi-headed attention based on "Attention
+  is all you Need". If `from_tensor` and `to_tensor` are the same, then
+  this is self-attention. Each timestep in `from_tensor` attends to the
+  corresponding sequence in `to_tensor`, and returns a fixed-with vector.
+
+  This function first projects `from_tensor` into a "query" tensor and
+  `to_tensor` into "key" and "value" tensors. These are (effectively) a list
+  of tensors of length `num_attention_heads`, where each tensor is of shape
+  [batch_size, seq_length, size_per_head].
+
+  Then, the query and key tensors are dot-producted and scaled. These are
+  softmaxed to obtain attention probabilities. The value tensors are then
+  interpolated by these probabilities, then concatenated back to a single
+  tensor and returned.
+
+  In practice, the multi-headed attention are done with transposes and
+  reshapes rather than actual separate tensors.
+
+  Args:
+    from_tensor: float Tensor of shape [batch_size, from_seq_length,
+      from_width].
+    to_tensor: float Tensor of shape [batch_size, to_seq_length, to_width].
+    attention_mask: (optional) int32 Tensor of shape [batch_size,
+      from_seq_length, to_seq_length]. The values should be 1 or 0. The
+      attention scores will effectively be set to -infinity for any positions in
+      the mask that are 0, and will be unchanged for positions that are 1.
+    num_attention_heads: int. Number of attention heads.
+    size_per_head: int. Size of each attention head.
+    query_act: (optional) Activation function for the query transform.
+    key_act: (optional) Activation function for the key transform.
+    value_act: (optional) Activation function for the value transform.
+    attention_probs_dropout_prob: (optional) float. Dropout probability of the
+      attention probabilities.
+    initializer_range: float. Range of the weight initializer.
+    do_return_2d_tensor: bool. If True, the output will be of shape [batch_size
+      * from_seq_length, num_attention_heads * size_per_head]. If False, the
+      output will be of shape [batch_size, from_seq_length, num_attention_heads
+      * size_per_head].
+    batch_size: (Optional) int. If the input is 2D, this might be the batch size
+      of the 3D version of the `from_tensor` and `to_tensor`.
+    from_seq_length: (Optional) If the input is 2D, this might be the seq length
+      of the 3D version of the `from_tensor`.
+    to_seq_length: (Optional) If the input is 2D, this might be the seq length
+      of the 3D version of the `to_tensor`.
+
+  Returns:
+    float Tensor of shape [batch_size, from_seq_length,
+      num_attention_heads * size_per_head]. (If `do_return_2d_tensor` is
+      true, this will be of shape [batch_size * from_seq_length,
+      num_attention_heads * size_per_head]).
+
+  Raises:
+    ValueError: Any of the arguments or tensor shapes are invalid.
+  """
+
+  def transpose_for_scores(input_tensor, batch_size, num_attention_heads,
+                           seq_length, width):
+    output_tensor = tf.reshape(
+        input_tensor, [batch_size, seq_length, num_attention_heads, width])
+
+    output_tensor = tf.transpose(a=output_tensor, perm=[0, 2, 1, 3])
+    return output_tensor
+
+  from_shape = get_shape_list(from_tensor, expected_rank=[2, 3])
+  to_shape = get_shape_list(to_tensor, expected_rank=[2, 3])
+
+  if len(from_shape) != len(to_shape):
+    raise ValueError(
+        "The rank of `from_tensor` must match the rank of `to_tensor`.")
+
+  if len(from_shape) == 3:
+    batch_size = from_shape[0]
+    from_seq_length = from_shape[1]
+    to_seq_length = to_shape[1]
+  elif len(from_shape) == 2:
+    if (batch_size is None or from_seq_length is None or to_seq_length is None):
+      raise ValueError(
+          "When passing in rank 2 tensors to attention_layer, the values "
+          "for `batch_size`, `from_seq_length`, and `to_seq_length` "
+          "must all be specified.")
+
+  # Scalar dimensions referenced here:
+  #   B = batch size (number of sequences)
+  #   F = `from_tensor` sequence length
+  #   T = `to_tensor` sequence length
+  #   N = `num_attention_heads`
+  #   H = `size_per_head`
+
+  from_tensor_2d = reshape_to_matrix(from_tensor)
+  to_tensor_2d = reshape_to_matrix(to_tensor)
+
+  # `query_layer` = [B*F, N*H]
+  query_layer = tf.compat.v1.layers.dense(
+      from_tensor_2d,
+      num_attention_heads * size_per_head,
+      activation=query_act,
+      name="query",
+      kernel_initializer=create_initializer(initializer_range))
+
+  # `key_layer` = [B*T, N*H]
+  key_layer = tf.compat.v1.layers.dense(
+      to_tensor_2d,
+      num_attention_heads * size_per_head,
+      activation=key_act,
+      name="key",
+      kernel_initializer=create_initializer(initializer_range))
+
+  # `value_layer` = [B*T, N*H]
+  value_layer = tf.compat.v1.layers.dense(
+      to_tensor_2d,
+      num_attention_heads * size_per_head,
+      activation=value_act,
+      name="value",
+      kernel_initializer=create_initializer(initializer_range))
+
+  # `query_layer` = [B, N, F, H]
+  query_layer = transpose_for_scores(query_layer, batch_size,
+                                     num_attention_heads, from_seq_length,
+                                     size_per_head)
+
+  # `key_layer` = [B, N, T, H]
+  key_layer = transpose_for_scores(key_layer, batch_size, num_attention_heads,
+                                   to_seq_length, size_per_head)
+
+  # Take the dot product between "query" and "key" to get the raw
+  # attention scores.
+  # `attention_scores` = [B, N, F, T]
+  attention_scores = tf.matmul(query_layer, key_layer, transpose_b=True)
+  attention_scores = tf.multiply(attention_scores, bf.r_cast(
+                                 1.0 / math.sqrt(float(size_per_head))))
+
+  if attention_mask is not None:
+    # `attention_mask` = [B, 1, F, T]
+    attention_mask = tf.expand_dims(attention_mask, axis=[1])
+
+    # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
+    # masked positions, this operation will create a tensor which is 0.0 for
+    # positions we want to attend and -10000.0 for masked positions.
+    adder = (1.0 - tf.cast(attention_mask, from_tensor.dtype)) * -10000.0
+
+    # Since we are adding it to the raw scores before the softmax, this is
+    # effectively the same as removing these entirely.
+    attention_scores += adder
+
+  # Normalize the attention scores to probabilities.
+  # `attention_probs` = [B, N, F, T]
+  attention_probs = bf.softmax(attention_scores)
+
+  # This is actually dropping out entire tokens to attend to, which might
+  # seem a bit unusual, but is taken from the original Transformer paper.
+  attention_probs = dropout(attention_probs, attention_probs_dropout_prob)
+
+  # `value_layer` = [B, T, N, H]
+  value_layer = tf.reshape(
+      value_layer,
+      [batch_size, to_seq_length, num_attention_heads, size_per_head])
+
+  # `value_layer` = [B, N, T, H]
+  value_layer = tf.transpose(a=value_layer, perm=[0, 2, 1, 3])
+
+  # `context_layer` = [B, N, F, H]
+  context_layer = tf.matmul(attention_probs, value_layer)
+
+  # `context_layer` = [B, F, N, H]
+  context_layer = tf.transpose(a=context_layer, perm=[0, 2, 1, 3])
+
+  if do_return_2d_tensor:
+    # `context_layer` = [B*F, N*H]
+    context_layer = tf.reshape(
+        context_layer,
+        [batch_size * from_seq_length, num_attention_heads * size_per_head])
+  else:
+    # `context_layer` = [B, F, N*H]
+    context_layer = tf.reshape(
+        context_layer,
+        [batch_size, from_seq_length, num_attention_heads * size_per_head])
+
+  return context_layer
+
+
+def transformer_model(input_tensor,
+                      attention_mask=None,
+                      hidden_size=768,
+                      num_hidden_layers=12,
+                      num_attention_heads=12,
+                      intermediate_size=3072,
+                      intermediate_act_fn=bf.gelu,
+                      hidden_dropout_prob=0.1,
+                      attention_probs_dropout_prob=0.1,
+                      initializer_range=0.02,
+                      do_return_all_layers=False):
+  """Multi-headed, multi-layer Transformer from "Attention is All You Need".
+
+  This is almost an exact implementation of the original Transformer encoder.
+
+  See the original paper:
+  https://arxiv.org/abs/1706.03762
+
+  Also see:
+  https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py
+
+  Args:
+    input_tensor: float Tensor of shape [batch_size, seq_length, hidden_size].
+    attention_mask: (optional) int32 Tensor of shape [batch_size, seq_length,
+      seq_length], with 1 for positions that can be attended to and 0 in
+      positions that should not be.
+    hidden_size: int. Hidden size of the Transformer.
+    num_hidden_layers: int. Number of layers (blocks) in the Transformer.
+    num_attention_heads: int. Number of attention heads in the Transformer.
+    intermediate_size: int. The size of the "intermediate" (a.k.a., feed
+      forward) layer.
+    intermediate_act_fn: function. The non-linear activation function to apply
+      to the output of the intermediate/feed-forward layer.
+    hidden_dropout_prob: float. Dropout probability for the hidden layers.
+    attention_probs_dropout_prob: float. Dropout probability of the attention
+      probabilities.
+    initializer_range: float. Range of the initializer (stddev of truncated
+      normal).
+    do_return_all_layers: Whether to also return all layers or just the final
+      layer.
+
+  Returns:
+    float Tensor of shape [batch_size, seq_length, hidden_size], the final
+    hidden layer of the Transformer.
+
+  Raises:
+    ValueError: A Tensor shape or parameter is invalid.
+  """
+  if hidden_size % num_attention_heads != 0:
+    raise ValueError(
+        "The hidden size (%d) is not a multiple of the number of attention "
+        "heads (%d)" % (hidden_size, num_attention_heads))
+
+  attention_head_size = int(hidden_size / num_attention_heads)
+  input_shape = get_shape_list(input_tensor, expected_rank=3)
+  batch_size = input_shape[0]
+  seq_length = input_shape[1]
+  input_width = input_shape[2]
+
+  # The Transformer performs sum residuals on all layers so the input needs
+  # to be the same as the hidden size.
+  if input_width != hidden_size:
+    raise ValueError("The width of the input tensor (%d) != hidden size (%d)" %
+                     (input_width, hidden_size))
+
+  # We keep the representation as a 2D tensor to avoid re-shaping it back and
+  # forth from a 3D tensor to a 2D tensor. Re-shapes are normally free on
+  # the GPU/CPU but may not be free on the TPU, so we want to minimize them to
+  # help the optimizer.
+  prev_output = reshape_to_matrix(input_tensor)
+
+  all_layer_outputs = []
+  for layer_idx in range(num_hidden_layers):
+    with tf.compat.v1.variable_scope("layer_%d" % layer_idx):
+      layer_input = prev_output
+
+      with tf.compat.v1.variable_scope("attention"):
+        attention_heads = []
+        with tf.compat.v1.variable_scope("self"):
+          attention_head = attention_layer(
+              from_tensor=layer_input,
+              to_tensor=layer_input,
+              attention_mask=attention_mask,
+              num_attention_heads=num_attention_heads,
+              size_per_head=attention_head_size,
+              attention_probs_dropout_prob=attention_probs_dropout_prob,
+              initializer_range=initializer_range,
+              do_return_2d_tensor=True,
+              batch_size=batch_size,
+              from_seq_length=seq_length,
+              to_seq_length=seq_length)
+          attention_heads.append(attention_head)
+
+        attention_output = None
+        if len(attention_heads) == 1:
+          attention_output = attention_heads[0]
+        else:
+          # In the case where we have other sequences, we just concatenate
+          # them to the self-attention head before the projection.
+          attention_output = tf.concat(attention_heads, axis=-1)
+
+        # Run a linear projection of `hidden_size` then add a residual
+        # with `layer_input`.
+        with tf.compat.v1.variable_scope("output"):
+          attention_output = tf.compat.v1.layers.dense(
+              attention_output,
+              hidden_size,
+              kernel_initializer=create_initializer(initializer_range))
+          attention_output = dropout(attention_output, hidden_dropout_prob)
+          attention_output = layer_norm(attention_output + layer_input)
+
+      # The activation is only applied to the "intermediate" hidden layer.
+      with tf.compat.v1.variable_scope("intermediate"):
+        intermediate_output = tf.compat.v1.layers.dense(
+            attention_output,
+            intermediate_size,
+            activation=intermediate_act_fn,
+            kernel_initializer=create_initializer(initializer_range))
+
+      # Down-project back to `hidden_size` then add the residual.
+      with tf.compat.v1.variable_scope("output"):
+        layer_output = tf.compat.v1.layers.dense(
+            intermediate_output,
+            hidden_size,
+            kernel_initializer=create_initializer(initializer_range))
+        layer_output = dropout(layer_output, hidden_dropout_prob)
+        layer_output = layer_norm(layer_output + attention_output)
+        prev_output = layer_output
+        all_layer_outputs.append(layer_output)
+
+  if do_return_all_layers:
+    final_outputs = []
+    for layer_output in all_layer_outputs:
+      final_output = reshape_from_matrix(layer_output, input_shape)
+      final_outputs.append(final_output)
+    return final_outputs
+  else:
+    final_output = reshape_from_matrix(prev_output, input_shape)
+    return final_output
+
+
+def get_shape_list(tensor, expected_rank=None, name=None):
+  """Returns a list of the shape of tensor, preferring static dimensions.
+
+  Args:
+    tensor: A tf.Tensor object to find the shape of.
+    expected_rank: (optional) int. The expected rank of `tensor`. If this is
+      specified and the `tensor` has a different rank, and exception will be
+      thrown.
+    name: Optional name of the tensor for the error message.
+
+  Returns:
+    A list of dimensions of the shape of tensor. All static dimensions will
+    be returned as python integers, and dynamic dimensions will be returned
+    as tf.Tensor scalars.
+  """
+  if name is None:
+    name = tensor.name
+
+  if expected_rank is not None:
+    assert_rank(tensor, expected_rank, name)
+
+  shape = tensor.shape.as_list()
+
+  non_static_indexes = []
+  for (index, dim) in enumerate(shape):
+    if dim is None:
+      non_static_indexes.append(index)
+
+  if not non_static_indexes:
+    return shape
+
+  dyn_shape = tf.shape(input=tensor)
+  for index in non_static_indexes:
+    shape[index] = dyn_shape[index]
+  return shape
+
+
+def reshape_to_matrix(input_tensor):
+  """Reshapes a >= rank 2 tensor to a rank 2 tensor (i.e., a matrix)."""
+  ndims = input_tensor.shape.ndims
+  if ndims < 2:
+    raise ValueError("Input tensor must have at least rank 2. Shape = %s" %
+                     (input_tensor.shape))
+  if ndims == 2:
+    return input_tensor
+
+  width = input_tensor.shape[-1]
+  output_tensor = tf.reshape(input_tensor, [-1, width])
+  return output_tensor
+
+
+def reshape_from_matrix(output_tensor, orig_shape_list):
+  """Reshapes a rank 2 tensor back to its original rank >= 2 tensor."""
+  if len(orig_shape_list) == 2:
+    return output_tensor
+
+  output_shape = get_shape_list(output_tensor)
+
+  orig_dims = orig_shape_list[0:-1]
+  width = output_shape[-1]
+
+  return tf.reshape(output_tensor, orig_dims + [width])
+
+
+def assert_rank(tensor, expected_rank, name=None):
+  """Raises an exception if the tensor rank is not of the expected rank.
+
+  Args:
+    tensor: A tf.Tensor to check the rank of.
+    expected_rank: Python integer or list of integers, expected rank.
+    name: Optional name of the tensor for the error message.
+
+  Raises:
+    ValueError: If the expected shape doesn't match the actual shape.
+  """
+  if name is None:
+    name = tensor.name
+
+  expected_rank_dict = {}
+  if isinstance(expected_rank, six.integer_types):
+    expected_rank_dict[expected_rank] = True
+  else:
+    for x in expected_rank:
+      expected_rank_dict[x] = True
+
+  actual_rank = tensor.shape.ndims
+  if actual_rank not in expected_rank_dict:
+    scope_name = tf.compat.v1.get_variable_scope().name
+    raise ValueError(
+        "For the tensor `%s` in scope `%s`, the actual rank "
+        "`%d` (shape = %s) is not equal to the expected rank `%s`" %
+        (name, scope_name, actual_rank, str(tensor.shape), str(expected_rank)))
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/requirements.txt b/models/language_modeling/tensorflow/bert_large/inference/gpu/requirements.txt
new file mode 100644
index 000000000..059fa43da
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/requirements.txt
@@ -0,0 +1,2 @@
+intel-tensorflow>=2.5.0 # CPU Version of TensorFlow.
+# tensorflow-gpu  >= 1.11.0 # GPU version of TensorFlow.
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/run_classifier.py b/models/language_modeling/tensorflow/bert_large/inference/gpu/run_classifier.py
new file mode 100644
index 000000000..e5f9f45fa
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/run_classifier.py
@@ -0,0 +1,1114 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import csv
+import os
+import modeling
+import optimization
+import tokenization
+import tensorflow as tf
+import generic_ops as bf
+
+from datetime import datetime
+from datetime import timedelta
+
+
+from absl import app
+#from absl import flags
+from absl import logging
+flags = tf.compat.v1.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+    "data_dir", None,
+    "The input data dir. Should contain the .tsv files (or other data files) "
+    "for the task.")
+
+flags.DEFINE_string(
+    "bert_config_file", None,
+    "The config json file corresponding to the pre-trained BERT model. "
+    "This specifies the model architecture.")
+
+flags.DEFINE_string("task_name", None, "The name of the task to train.")
+
+flags.DEFINE_string("vocab_file", None,
+                    "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_string(
+    "output_dir", None,
+    "The output directory where the model checkpoints will be written.")
+
+## Other parameters
+
+flags.DEFINE_string(
+    "init_checkpoint", None,
+    "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_bool(
+    "do_lower_case", True,
+    "Whether to lower case the input text. Should be True for uncased "
+    "models and False for cased models.")
+
+flags.DEFINE_integer(
+    "max_seq_length", 128,
+    "The maximum total input sequence length after WordPiece tokenization. "
+    "Sequences longer than this will be truncated, and sequences shorter "
+    "than this will be padded.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_eval", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_bool(
+    "do_predict", False,
+    "Whether to run the model in inference mode on the test set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("accum_steps", 1, "Accumulation steps for batch size greater than 32.")
+
+flags.DEFINE_integer("eval_batch_size", 8, "Total batch size for eval.")
+
+flags.DEFINE_integer("predict_batch_size", 8, "Total batch size for predict.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_float("num_train_epochs", 3.0,
+                   "Total number of training epochs to perform.")
+
+flags.DEFINE_float(
+    "warmup_proportion", 0.1,
+    "Proportion of training to perform linear learning rate warmup for. "
+    "E.g., 0.1 = 10% of training.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+                     "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+                     "How many steps to make in each estimator call.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+flags.DEFINE_string(
+    "tpu_name", None,
+    "The Cloud TPU to use for training. This should be either the name "
+    "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+    "url.")
+
+flags.DEFINE_string(
+    "tpu_zone", None,
+    "[Optional] GCE zone where the Cloud TPU is located in. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string(
+    "gcp_project", None,
+    "[Optional] Project name for the Cloud TPU-enabled project. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+
+flags.DEFINE_integer(
+    "num_tpu_cores", 8,
+    "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+flags.DEFINE_integer( "inter_op_parallelism_threads", 2,
+                      "Setting inter op for the model")
+
+flags.DEFINE_integer("intra_op_parallelism_threads", 27,
+                     "Setting inter op for the model")
+
+flags.DEFINE_bool("profile", False, "[Optional] To enable Tensorflow profile hook."
+                                    "The profile output will be generated in the output_dir")
+
+flags.DEFINE_string("precision", "fp32", "[Optional] TensorFlow training precision.")
+
+flags.DEFINE_string("frozen_graph_path", None, "path of frozen graph.")
+
+flags.DEFINE_bool("experimental_gelu", False,
+    "[Optional] If true, use experimental gelu op in model."
+    "           Be careful this flag will crash model with incompatible TF.")
+
+flags.DEFINE_bool("optimized_softmax", False,
+    "[Optional] If true, use optimized softmax op in model.")
+
+class LoggerHook(tf.estimator.SessionRunHook):
+  """ Logs runtime. """
+
+  def __init__(self, batch_size):
+    self.batch_size = batch_size
+
+  def begin(self):
+    self._step = 0
+    self._total_duration = 0
+    self._warmup = 2
+
+  def before_run(self, run_context):
+    self._start_time = datetime.now()
+
+  def after_run(self, run_context, run_values):
+    self._step += 1
+    duration = datetime.now() - self._start_time
+    ms = duration.total_seconds() * 1000.00
+    if self._step > self._warmup:
+      self._total_duration += ms
+      #if self._step % 100 == 0:
+      print("Current step: %d, time in ms: %.2f" %(self._step, ms))
+      time_takes = self._total_duration / (self._step - self._warmup)
+      print("Throughput is %.2f samples/s" % (self.batch_size * 1000 / time_takes))
+    else:
+      print("Warmup step: %d, time in ms: %.2f" %(self._step, ms))
+
+  def end(self, run_context):
+    print("self._step: %d" %self._step)
+    print("Total time spent (after warmup): %.2f ms" %(self._total_duration))
+    print("Time spent per iteration (after warmup): %.2f ms" %(self._total_duration/(self._step - self._warmup)))
+    time_takes = self._total_duration / (self._step - self._warmup)
+    if self.batch_size == 1:
+      print("Latency is %.2f ms" % (time_takes))
+    print("Throughput is %.2f samples/s" % (self.batch_size * 1000 / time_takes))
+
+class InputExample(object):
+  """A single training/test example for simple sequence classification."""
+
+  def __init__(self, guid, text_a, text_b=None, label=None):
+    """Constructs a InputExample.
+    Args:
+      guid: Unique id for the example.
+      text_a: string. The untokenized text of the first sequence. For single
+        sequence tasks, only this sequence must be specified.
+      text_b: (Optional) string. The untokenized text of the second sequence.
+        Only must be specified for sequence pair tasks.
+      label: (Optional) string. The label of the example. This should be
+        specified for train and dev examples, but not for test examples.
+    """
+    self.guid = guid
+    self.text_a = text_a
+    self.text_b = text_b
+    self.label = label
+
+
+class PaddingInputExample(object):
+  """Fake example so the num input examples is a multiple of the batch size.
+  When running eval/predict on the TPU, we need to pad the number of examples
+  to be a multiple of the batch size, because the TPU requires a fixed batch
+  size. The alternative is to drop the last batch, which is bad because it means
+  the entire output data won't be generated.
+  We use this class instead of `None` because treating `None` as padding
+  battches could cause silent errors.
+  """
+
+
+class InputFeatures(object):
+  """A single set of features of data."""
+
+  def __init__(self,
+               input_ids,
+               input_mask,
+               segment_ids,
+               label_id,
+               is_real_example=True):
+    self.input_ids = input_ids
+    self.input_mask = input_mask
+    self.segment_ids = segment_ids
+    self.label_id = label_id
+    self.is_real_example = is_real_example
+
+
+class DataProcessor(object):
+  """Base class for data converters for sequence classification data sets."""
+
+  def get_train_examples(self, data_dir):
+    """Gets a collection of `InputExample`s for the train set."""
+    raise NotImplementedError()
+
+  def get_dev_examples(self, data_dir):
+    """Gets a collection of `InputExample`s for the dev set."""
+    raise NotImplementedError()
+
+  def get_test_examples(self, data_dir):
+    """Gets a collection of `InputExample`s for prediction."""
+    raise NotImplementedError()
+
+  def get_labels(self):
+    """Gets the list of labels for this data set."""
+    raise NotImplementedError()
+
+  @classmethod
+  def _read_tsv(cls, input_file, quotechar=None):
+    """Reads a tab separated value file."""
+    with tf.io.gfile.GFile(input_file, "r") as f:
+      reader = csv.reader(f, delimiter="\t", quotechar=quotechar)
+      lines = []
+      for line in reader:
+        lines.append(line)
+      return lines
+
+
+class XnliProcessor(DataProcessor):
+  """Processor for the XNLI data set."""
+
+  def __init__(self):
+    self.language = "zh"
+
+  def get_train_examples(self, data_dir):
+    """See base class."""
+    lines = self._read_tsv(
+        os.path.join(data_dir, "multinli",
+                     "multinli.train.%s.tsv" % self.language))
+    examples = []
+    for (i, line) in enumerate(lines):
+      if i == 0:
+        continue
+      guid = "train-%d" % (i)
+      text_a = tokenization.convert_to_unicode(line[0])
+      text_b = tokenization.convert_to_unicode(line[1])
+      label = tokenization.convert_to_unicode(line[2])
+      if label == tokenization.convert_to_unicode("contradictory"):
+        label = tokenization.convert_to_unicode("contradiction")
+      examples.append(
+          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+    return examples
+
+  def get_dev_examples(self, data_dir):
+    """See base class."""
+    lines = self._read_tsv(os.path.join(data_dir, "xnli.dev.tsv"))
+    examples = []
+    for (i, line) in enumerate(lines):
+      if i == 0:
+        continue
+      guid = "dev-%d" % (i)
+      language = tokenization.convert_to_unicode(line[0])
+      if language != tokenization.convert_to_unicode(self.language):
+        continue
+      text_a = tokenization.convert_to_unicode(line[6])
+      text_b = tokenization.convert_to_unicode(line[7])
+      label = tokenization.convert_to_unicode(line[1])
+      examples.append(
+          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+    return examples
+
+  def get_labels(self):
+    """See base class."""
+    return ["contradiction", "entailment", "neutral"]
+
+
+class MnliProcessor(DataProcessor):
+  """Processor for the MultiNLI data set (GLUE version)."""
+
+  def get_train_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+  def get_dev_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "dev_matched.tsv")),
+        "dev_matched")
+
+  def get_test_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "test_matched.tsv")), "test")
+
+  def get_labels(self):
+    """See base class."""
+    return ["contradiction", "entailment", "neutral"]
+
+  def _create_examples(self, lines, set_type):
+    """Creates examples for the training and dev sets."""
+    examples = []
+    for (i, line) in enumerate(lines):
+      if i == 0:
+        continue
+      guid = "%s-%s" % (set_type, tokenization.convert_to_unicode(line[0]))
+      text_a = tokenization.convert_to_unicode(line[8])
+      text_b = tokenization.convert_to_unicode(line[9])
+      if set_type == "test":
+        label = "contradiction"
+      else:
+        label = tokenization.convert_to_unicode(line[-1])
+      examples.append(
+          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+    return examples
+
+
+class MrpcProcessor(DataProcessor):
+  """Processor for the MRPC data set (GLUE version)."""
+
+  def get_train_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+  def get_dev_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+  def get_test_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")
+
+  def get_labels(self):
+    """See base class."""
+    return ["0", "1"]
+
+  def _create_examples(self, lines, set_type):
+    """Creates examples for the training and dev sets."""
+    examples = []
+    for (i, line) in enumerate(lines):
+      if i == 0:
+        continue
+      guid = "%s-%s" % (set_type, i)
+      text_a = tokenization.convert_to_unicode(line[3])
+      text_b = tokenization.convert_to_unicode(line[4])
+      if set_type == "test":
+        label = "0"
+      else:
+        label = tokenization.convert_to_unicode(line[0])
+      examples.append(
+          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+    return examples
+
+
+class ColaProcessor(DataProcessor):
+  """Processor for the CoLA data set (GLUE version)."""
+
+  def get_train_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+  def get_dev_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+  def get_test_examples(self, data_dir):
+    """See base class."""
+    return self._create_examples(
+        self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")
+
+  def get_labels(self):
+    """See base class."""
+    return ["0", "1"]
+
+  def _create_examples(self, lines, set_type):
+    """Creates examples for the training and dev sets."""
+    examples = []
+    for (i, line) in enumerate(lines):
+      # Only the test set has a header
+      if set_type == "test" and i == 0:
+        continue
+      guid = "%s-%s" % (set_type, i)
+      if set_type == "test":
+        text_a = tokenization.convert_to_unicode(line[1])
+        label = "0"
+      else:
+        text_a = tokenization.convert_to_unicode(line[3])
+        label = tokenization.convert_to_unicode(line[1])
+      examples.append(
+          InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
+    return examples
+
+
+def convert_single_example(ex_index, example, label_list, max_seq_length,
+                           tokenizer):
+  """Converts a single `InputExample` into a single `InputFeatures`."""
+
+  if isinstance(example, PaddingInputExample):
+    return InputFeatures(
+        input_ids=[0] * max_seq_length,
+        input_mask=[0] * max_seq_length,
+        segment_ids=[0] * max_seq_length,
+        label_id=0,
+        is_real_example=False)
+
+  label_map = {}
+  for (i, label) in enumerate(label_list):
+    label_map[label] = i
+
+  tokens_a = tokenizer.tokenize(example.text_a)
+  tokens_b = None
+  if example.text_b:
+    tokens_b = tokenizer.tokenize(example.text_b)
+
+  if tokens_b:
+    # Modifies `tokens_a` and `tokens_b` in place so that the total
+    # length is less than the specified length.
+    # Account for [CLS], [SEP], [SEP] with "- 3"
+    _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
+  else:
+    # Account for [CLS] and [SEP] with "- 2"
+    if len(tokens_a) > max_seq_length - 2:
+      tokens_a = tokens_a[0:(max_seq_length - 2)]
+
+  # The convention in BERT is:
+  # (a) For sequence pairs:
+  #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+  #  type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1
+  # (b) For single sequences:
+  #  tokens:   [CLS] the dog is hairy . [SEP]
+  #  type_ids: 0     0   0   0  0     0 0
+  #
+  # Where "type_ids" are used to indicate whether this is the first
+  # sequence or the second sequence. The embedding vectors for `type=0` and
+  # `type=1` were learned during pre-training and are added to the wordpiece
+  # embedding vector (and position vector). This is not *strictly* necessary
+  # since the [SEP] token unambiguously separates the sequences, but it makes
+  # it easier for the model to learn the concept of sequences.
+  #
+  # For classification tasks, the first vector (corresponding to [CLS]) is
+  # used as the "sentence vector". Note that this only makes sense because
+  # the entire model is fine-tuned.
+  tokens = []
+  segment_ids = []
+  tokens.append("[CLS]")
+  segment_ids.append(0)
+  for token in tokens_a:
+    tokens.append(token)
+    segment_ids.append(0)
+  tokens.append("[SEP]")
+  segment_ids.append(0)
+
+  if tokens_b:
+    for token in tokens_b:
+      tokens.append(token)
+      segment_ids.append(1)
+    tokens.append("[SEP]")
+    segment_ids.append(1)
+
+  input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+  # The mask has 1 for real tokens and 0 for padding tokens. Only real
+  # tokens are attended to.
+  input_mask = [1] * len(input_ids)
+
+  # Zero-pad up to the sequence length.
+  while len(input_ids) < max_seq_length:
+    input_ids.append(0)
+    input_mask.append(0)
+    segment_ids.append(0)
+
+  assert len(input_ids) == max_seq_length
+  assert len(input_mask) == max_seq_length
+  assert len(segment_ids) == max_seq_length
+
+  label_id = label_map[example.label]
+  if ex_index < 5:
+    tf.compat.v1.logging.info("*** Example ***")
+    tf.compat.v1.logging.info("guid: %s" % (example.guid))
+    tf.compat.v1.logging.info("tokens: %s" % " ".join(
+        [tokenization.printable_text(x) for x in tokens]))
+    tf.compat.v1.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+    tf.compat.v1.logging.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+    tf.compat.v1.logging.info("segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+    tf.compat.v1.logging.info("label: %s (id = %d)" % (example.label, label_id))
+
+  feature = InputFeatures(
+      input_ids=input_ids,
+      input_mask=input_mask,
+      segment_ids=segment_ids,
+      label_id=label_id,
+      is_real_example=True)
+  return feature
+
+
+def file_based_convert_examples_to_features(
+    examples, label_list, max_seq_length, tokenizer, output_file):
+  """Convert a set of `InputExample`s to a TFRecord file."""
+
+  writer = tf.io.TFRecordWriter(output_file)
+
+  for (ex_index, example) in enumerate(examples):
+    if ex_index % 10000 == 0:
+      tf.compat.v1.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
+
+    feature = convert_single_example(ex_index, example, label_list,
+                                     max_seq_length, tokenizer)
+
+    def create_int_feature(values):
+      f = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))
+      return f
+
+    features = collections.OrderedDict()
+    features["input_ids"] = create_int_feature(feature.input_ids)
+    features["input_mask"] = create_int_feature(feature.input_mask)
+    features["segment_ids"] = create_int_feature(feature.segment_ids)
+    features["label_ids"] = create_int_feature([feature.label_id])
+    features["is_real_example"] = create_int_feature(
+        [int(feature.is_real_example)])
+
+    tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+    writer.write(tf_example.SerializeToString())
+  writer.close()
+
+
+def file_based_input_fn_builder(input_file, seq_length, is_training,
+                                drop_remainder):
+  """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+  name_to_features = {
+      "input_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "input_mask": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "segment_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "label_ids": tf.io.FixedLenFeature([], tf.int64),
+      "is_real_example": tf.io.FixedLenFeature([], tf.int64),
+  }
+
+  def _decode_record(record, name_to_features):
+    """Decodes a record to a TensorFlow example."""
+    example = tf.io.parse_single_example(serialized=record, features=name_to_features)
+
+    # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+    # So cast all int64 to int32.
+    for name in list(example.keys()):
+      t = example[name]
+      if t.dtype == tf.int64:
+        t = tf.cast(t, dtype=tf.int32)
+      example[name] = t
+
+    return example
+
+  def input_fn(params):
+    """The actual input function."""
+    batch_size = params["batch_size"]
+
+    # For training, we want a lot of parallel reading and shuffling.
+    # For eval, we want no shuffling and parallel reading doesn't matter.
+    d = tf.data.TFRecordDataset(input_file)
+    if is_training:
+      d = d.repeat()
+      d = d.shuffle(buffer_size=100)
+
+    d = d.apply(
+        tf.data.experimental.map_and_batch(
+            lambda record: _decode_record(record, name_to_features),
+            batch_size=batch_size,
+            drop_remainder=drop_remainder))
+
+    return d
+
+  return input_fn
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+  """Truncates a sequence pair in place to the maximum length."""
+
+  # This is a simple heuristic which will always truncate the longer sequence
+  # one token at a time. This makes more sense than truncating an equal percent
+  # of tokens from each, since if one sequence is very short then each token
+  # that's truncated likely contains more information than a longer sequence.
+  while True:
+    total_length = len(tokens_a) + len(tokens_b)
+    if total_length <= max_length:
+      break
+    if len(tokens_a) > len(tokens_b):
+      tokens_a.pop()
+    else:
+      tokens_b.pop()
+
+
+def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
+                 labels, num_labels, use_one_hot_embeddings):
+  """Creates a classification model."""
+  model = modeling.BertModel(
+      config=bert_config,
+      is_training=is_training,
+      input_ids=input_ids,
+      input_mask=input_mask,
+      token_type_ids=segment_ids,
+      use_one_hot_embeddings=use_one_hot_embeddings)
+
+  # In the demo, we are doing a simple classification task on the entire
+  # segment.
+  #
+  # If you want to use the token-level output, use model.get_sequence_output()
+  # instead.
+  output_layer = model.get_pooled_output()
+
+  #hidden_size = output_layer.shape[-1].value
+  # For V2 value does not work
+  hidden_size = output_layer.shape[-1]
+
+  output_weights = tf.compat.v1.get_variable(
+      "cls/classifier/output_weights", [num_labels, hidden_size],
+      initializer=tf.compat.v1.truncated_normal_initializer(stddev=0.02))
+
+  output_bias = tf.compat.v1.get_variable(
+      "cls/classifier/output_bias", [num_labels], initializer=tf.compat.v1.zeros_initializer())
+
+  with tf.compat.v1.variable_scope("loss"):
+    if is_training:
+      # I.e., 0.1 dropout
+      output_layer = tf.nn.dropout(output_layer, rate=1 - (0.9))
+
+    logits = tf.matmul(output_layer, output_weights, transpose_b=True)
+    logits = tf.nn.bias_add(logits, output_bias)
+    probabilities = bf.softmax(logits, axis=-1)
+    log_probs = tf.nn.log_softmax(logits, axis=-1)
+
+    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
+
+    per_example_loss = -tf.reduce_sum(input_tensor=one_hot_labels * log_probs, axis=-1)
+    loss = tf.reduce_mean(input_tensor=per_example_loss)
+
+    return (loss, per_example_loss, logits, probabilities)
+
+
+def create_model_top(bert_config, is_training, input_ids, input_mask, segment_ids,
+                     label_ids, num_labels, use_one_hot_embeddings, frozen_graph=None):
+
+    if frozen_graph and not is_training:
+        inputs ={'input_ids': input_ids,
+                 'input_mask': input_mask,
+                 'segment_ids': segment_ids,
+                 'label_ids': label_ids}
+
+        outputs = ['loss/Mean', 'loss/Sum', 'loss/BiasAdd', 'loss/Softmax']
+        outputs = ['{}:0'.format(name) for name in outputs]
+
+        with tf.io.gfile.GFile(frozen_graph, "rb") as f:
+            graph_def = tf.compat.v1.GraphDef()
+            graph_def.ParseFromString(f.read())
+
+        return tf.graph_util.import_graph_def(graph_def, inputs, outputs, name="")
+
+    if bert_config.precision == "bfloat16" :
+      with tf.compat.v1.tpu.bfloat16_scope():
+        (total_loss, per_example_loss, logits, probabilities) = create_model(
+          bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
+          num_labels, use_one_hot_embeddings)
+        logits = bf.r_cast(logits)
+        return (total_loss, per_example_loss, logits, probabilities)
+    else :
+      (total_loss, per_example_loss, logits, probabilities) = create_model(
+        bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
+        num_labels, use_one_hot_embeddings)
+      return (total_loss, per_example_loss, logits, probabilities)
+
+def model_fn_builder(bert_config, num_labels, init_checkpoint, learning_rate,
+                     num_train_steps, num_warmup_steps, use_tpu,
+                     use_one_hot_embeddings, frozen_graph):
+  """Returns `model_fn` closure for TPUEstimator."""
+
+  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
+    """The `model_fn` for TPUEstimator."""
+
+    tf.compat.v1.logging.info("*** Features ***")
+    for name in sorted(features.keys()):
+      tf.compat.v1.logging.info("  name = %s, shape = %s" % (name, features[name].shape))
+
+    input_ids = features["input_ids"]
+    input_mask = features["input_mask"]
+    segment_ids = features["segment_ids"]
+    label_ids = features["label_ids"]
+    is_real_example = None
+    if "is_real_example" in features:
+      is_real_example = tf.cast(features["is_real_example"], dtype=tf.float32)
+    else:
+      is_real_example = tf.ones(tf.shape(input=label_ids), dtype=tf.float32)
+
+    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+    (total_loss, per_example_loss, logits, probabilities) = create_model_top(
+        bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
+        num_labels, use_one_hot_embeddings, frozen_graph)
+
+    tvars = tf.compat.v1.trainable_variables()
+    initialized_variable_names = {}
+    scaffold_fn = None
+    if init_checkpoint:
+      (assignment_map, initialized_variable_names
+      ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint, "Classifier")
+      if use_tpu:
+
+        def tpu_scaffold():
+          tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+          return tf.compat.v1.train.Scaffold()
+
+        scaffold_fn = tpu_scaffold
+      else:
+        tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+    tf.compat.v1.logging.info("**** Trainable Variables ****")
+    for var in tvars:
+      init_string = ""
+      if var.name in initialized_variable_names:
+        init_string = ", *INIT_FROM_CKPT*"
+      tf.compat.v1.logging.info("  name = %s, shape = %s%s", var.name, var.shape,
+                      init_string)
+
+    output_spec = None
+    if mode == tf.estimator.ModeKeys.TRAIN:
+
+      train_op = optimization.create_optimizer(
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu)
+
+      log_hook = bf.logTheLossHook(total_loss, n=10)
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode,
+          loss=total_loss,
+          train_op=train_op,
+          training_hooks=[log_hook],
+          scaffold_fn=scaffold_fn)
+    elif mode == tf.estimator.ModeKeys.EVAL:
+
+      def metric_fn(per_example_loss, label_ids, logits, is_real_example):
+        predictions = tf.argmax(input=logits, axis=-1, output_type=tf.int32)
+        accuracy = tf.compat.v1.metrics.accuracy(
+            labels=label_ids, predictions=predictions, weights=is_real_example)
+        loss = tf.compat.v1.metrics.mean(values=per_example_loss, weights=is_real_example)
+        return {
+            "eval_accuracy": accuracy,
+            "eval_loss": loss,
+        }
+
+      eval_metrics = (metric_fn,
+                      [per_example_loss, label_ids, logits, is_real_example])
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode,
+          loss=total_loss,
+          eval_metrics=eval_metrics,
+          scaffold_fn=scaffold_fn)
+    else:
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode,
+          predictions={"probabilities": probabilities},
+          scaffold_fn=scaffold_fn)
+    return output_spec
+
+  return model_fn
+
+
+# This function is not used by this file but is still used by the Colab and
+# people who depend on it.
+def input_fn_builder(features, seq_length, is_training, drop_remainder):
+  """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+  all_input_ids = []
+  all_input_mask = []
+  all_segment_ids = []
+  all_label_ids = []
+
+  for feature in features:
+    all_input_ids.append(feature.input_ids)
+    all_input_mask.append(feature.input_mask)
+    all_segment_ids.append(feature.segment_ids)
+    all_label_ids.append(feature.label_id)
+
+  def input_fn(params):
+    """The actual input function."""
+    batch_size = params["batch_size"]
+
+    num_examples = len(features)
+
+    # This is for demo purposes and does NOT scale to large data sets. We do
+    # not use Dataset.from_generator() because that uses tf.py_func which is
+    # not TPU compatible. The right way to load data is with TFRecordReader.
+    d = tf.data.Dataset.from_tensor_slices({
+        "input_ids":
+            tf.constant(
+                all_input_ids, shape=[num_examples, seq_length],
+                dtype=tf.int32),
+        "input_mask":
+            tf.constant(
+                all_input_mask,
+                shape=[num_examples, seq_length],
+                dtype=tf.int32),
+        "segment_ids":
+            tf.constant(
+                all_segment_ids,
+                shape=[num_examples, seq_length],
+                dtype=tf.int32),
+        "label_ids":
+            tf.constant(all_label_ids, shape=[num_examples], dtype=tf.int32),
+    })
+
+    if is_training:
+      d = d.repeat()
+      d = d.shuffle(buffer_size=100)
+
+    d = d.batch(batch_size=batch_size, drop_remainder=drop_remainder)
+    return d
+
+  return input_fn
+
+
+# This function is not used by this file but is still used by the Colab and
+# people who depend on it.
+def convert_examples_to_features(examples, label_list, max_seq_length,
+                                 tokenizer):
+  """Convert a set of `InputExample`s to a list of `InputFeatures`."""
+
+  features = []
+  for (ex_index, example) in enumerate(examples):
+    if ex_index % 10000 == 0:
+      tf.compat.v1.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
+
+    feature = convert_single_example(ex_index, example, label_list,
+                                     max_seq_length, tokenizer)
+
+    features.append(feature)
+  return features
+
+
+def main(_):
+  tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
+
+  if (FLAGS.accum_steps >1 ):
+    tf.compat.v1.logging.info(" Accum steps not yet supported in Classifier")
+    exit(0)
+  if FLAGS.profile:
+    tf.compat.v1.disable_eager_execution()
+
+  processors = {
+      "cola": ColaProcessor,
+      "mnli": MnliProcessor,
+      "mrpc": MrpcProcessor,
+      "xnli": XnliProcessor,
+  }
+
+  tokenization.validate_case_matches_checkpoint(FLAGS.do_lower_case,
+                                                FLAGS.init_checkpoint)
+
+  if not FLAGS.do_train and not FLAGS.do_eval and not FLAGS.do_predict:
+    raise ValueError(
+        "At least one of `do_train`, `do_eval` or `do_predict' must be True.")
+
+  bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+  bert_config.experimental_gelu = FLAGS.experimental_gelu
+
+  if FLAGS.precision:
+    bert_config.precision = FLAGS.precision
+
+  if FLAGS.max_seq_length > bert_config.max_position_embeddings:
+    raise ValueError(
+        "Cannot use sequence length %d because the BERT model "
+        "was only trained up to sequence length %d" %
+        (FLAGS.max_seq_length, bert_config.max_position_embeddings))
+
+  tf.io.gfile.makedirs(FLAGS.output_dir)
+
+  task_name = FLAGS.task_name.lower()
+
+  if task_name not in processors:
+    raise ValueError("Task not found: %s" % (task_name))
+
+  processor = processors[task_name]()
+
+  label_list = processor.get_labels()
+
+  tokenizer = tokenization.FullTokenizer(
+      vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+  tpu_cluster_resolver = None
+  if FLAGS.use_tpu and FLAGS.tpu_name:
+    tpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
+        FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+  session_config = tf.compat.v1.ConfigProto(
+      inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
+      intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads,
+      allow_soft_placement=True)
+
+  is_per_host = tf.compat.v1.estimator.tpu.InputPipelineConfig.PER_HOST_V2
+  run_config = tf.compat.v1.estimator.tpu.RunConfig(
+      cluster=tpu_cluster_resolver,
+      master=FLAGS.master,
+      model_dir=FLAGS.output_dir,
+      save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+      session_config=session_config,
+      tpu_config=tf.compat.v1.estimator.tpu.TPUConfig(
+          iterations_per_loop=FLAGS.iterations_per_loop,
+          num_shards=FLAGS.num_tpu_cores,
+          per_host_input_for_training=is_per_host))
+
+  train_examples = None
+  num_train_steps = None
+  num_warmup_steps = None
+  if FLAGS.do_train:
+    train_examples = processor.get_train_examples(FLAGS.data_dir)
+    num_train_steps = int(
+        len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+    num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+  model_fn = model_fn_builder(
+      bert_config=bert_config,
+      num_labels=len(label_list),
+      init_checkpoint=FLAGS.init_checkpoint,
+      learning_rate=FLAGS.learning_rate,
+      num_train_steps=num_train_steps,
+      num_warmup_steps=num_warmup_steps,
+      use_tpu=FLAGS.use_tpu,
+      use_one_hot_embeddings=FLAGS.use_tpu,
+      frozen_graph=FLAGS.frozen_graph_path)
+
+  # If TPU is not available, this will fall back to normal Estimator on CPU
+  # or GPU.
+  estimator = tf.compat.v1.estimator.tpu.TPUEstimator(
+      use_tpu=FLAGS.use_tpu,
+      model_fn=model_fn,
+      config=run_config,
+      train_batch_size=FLAGS.train_batch_size,
+      eval_batch_size=FLAGS.eval_batch_size,
+      predict_batch_size=FLAGS.predict_batch_size)
+
+  if FLAGS.do_train:
+    train_file = os.path.join(FLAGS.output_dir, "train.tf_record")
+    file_based_convert_examples_to_features(
+        train_examples, label_list, FLAGS.max_seq_length, tokenizer, train_file)
+    tf.compat.v1.logging.info("***** Running training *****")
+    tf.compat.v1.logging.info("  Num examples = %d", len(train_examples))
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.train_batch_size)
+    tf.compat.v1.logging.info("  Num steps = %d", num_train_steps)
+    train_input_fn = file_based_input_fn_builder(
+        input_file=train_file,
+        seq_length=FLAGS.max_seq_length,
+        is_training=True,
+        drop_remainder=True)
+
+    # Added to profile the run
+    if FLAGS.profile == True :
+      tf.compat.v1.logging.info("***** Running training with profiler *****")
+      hooks = [tf.compat.v1.train.ProfilerHook(save_steps=3, output_dir=FLAGS.output_dir,
+                                               show_memory=True)]
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps, hooks=hooks)
+    else:
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+  if FLAGS.do_eval:
+    eval_examples = processor.get_dev_examples(FLAGS.data_dir)
+    num_actual_eval_examples = len(eval_examples)
+    if FLAGS.use_tpu:
+      # TPU requires a fixed batch size for all batches, therefore the number
+      # of examples must be a multiple of the batch size, or else examples
+      # will get dropped. So we pad with fake examples which are ignored
+      # later on. These do NOT count towards the metric (all tf.metrics
+      # support a per-instance weight, and these get a weight of 0.0).
+      while len(eval_examples) % FLAGS.eval_batch_size != 0:
+        eval_examples.append(PaddingInputExample())
+
+    eval_file = os.path.join(FLAGS.output_dir, "eval.tf_record")
+    file_based_convert_examples_to_features(
+        eval_examples, label_list, FLAGS.max_seq_length, tokenizer, eval_file)
+
+    tf.compat.v1.logging.info("***** Running evaluation *****")
+    tf.compat.v1.logging.info("  Num examples = %d (%d actual, %d padding)",
+                    len(eval_examples), num_actual_eval_examples,
+                    len(eval_examples) - num_actual_eval_examples)
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.eval_batch_size)
+
+    # This tells the estimator to run through the entire set.
+    eval_steps = None
+    # However, if running eval on the TPU, you will need to specify the
+    # number of steps.
+    if FLAGS.use_tpu:
+      assert len(eval_examples) % FLAGS.eval_batch_size == 0
+      eval_steps = int(len(eval_examples) // FLAGS.eval_batch_size)
+
+    eval_drop_remainder = True if FLAGS.use_tpu else False
+    eval_input_fn = file_based_input_fn_builder(
+        input_file=eval_file,
+        seq_length=FLAGS.max_seq_length,
+        is_training=False,
+        drop_remainder=eval_drop_remainder)
+
+    hooks = [LoggerHook(FLAGS.eval_batch_size)]
+    if FLAGS.profile:
+      tf.compat.v1.logging.info("***** Running prediction with profiler*****")
+      hooks = [tf.estimator.ProfilerHook(save_steps=10, output_dir=FLAGS.output_dir,
+                                         show_memory=False)]
+
+    result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps, hooks=hooks)
+
+    output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt")
+    with tf.io.gfile.GFile(output_eval_file, "w") as writer:
+      tf.compat.v1.logging.info("***** Eval results *****")
+      for key in sorted(result.keys()):
+        tf.compat.v1.logging.info("  %s = %s", key, str(result[key]))
+        writer.write("%s = %s\n" % (key, str(result[key])))
+
+  if FLAGS.do_predict:
+    predict_examples = processor.get_test_examples(FLAGS.data_dir)
+    num_actual_predict_examples = len(predict_examples)
+    #if FLAGS.use_tpu:
+      # TPU requires a fixed batch size for all batches, therefore the number
+      # of examples must be a multiple of the batch size, or else examples
+      # will get dropped. So we pad with fake examples which are ignored
+      # later on.
+    while len(predict_examples) % FLAGS.predict_batch_size != 0:
+      predict_examples.append(PaddingInputExample())
+
+    predict_file = os.path.join(FLAGS.output_dir, "predict.tf_record")
+    file_based_convert_examples_to_features(predict_examples, label_list,
+                                            FLAGS.max_seq_length, tokenizer,
+                                            predict_file)
+
+    tf.compat.v1.logging.info("***** Running prediction*****")
+    tf.compat.v1.logging.info("  Num examples = %d (%d actual, %d padding)",
+                    len(predict_examples), num_actual_predict_examples,
+                    len(predict_examples) - num_actual_predict_examples)
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.predict_batch_size)
+
+    predict_drop_remainder = True if FLAGS.use_tpu else False
+    predict_input_fn = file_based_input_fn_builder(
+        input_file=predict_file,
+        seq_length=FLAGS.max_seq_length,
+        is_training=False,
+        drop_remainder=predict_drop_remainder)
+
+    hooks = [LoggerHook(FLAGS.predict_batch_size)]
+    if FLAGS.profile:
+      tf.compat.v1.logging.info("***** Running prediction with profiler*****")
+      hooks = [tf.estimator.ProfilerHook(save_steps=10, output_dir=FLAGS.output_dir,
+                                         show_memory=False)]
+    result = estimator.predict(input_fn=predict_input_fn, hooks=hooks)
+
+    output_predict_file = os.path.join(FLAGS.output_dir, "test_results.tsv")
+    with tf.io.gfile.GFile(output_predict_file, "w") as writer:
+      num_written_lines = 0
+      tf.compat.v1.logging.info("***** Predict results *****")
+      for (i, prediction) in enumerate(result):
+        probabilities = prediction["probabilities"]
+        if i >= num_actual_predict_examples:
+          break
+        output_line = "\t".join(
+            str(class_probability)
+            for class_probability in probabilities) + "\n"
+        writer.write(output_line)
+        num_written_lines += 1
+    assert num_written_lines == num_actual_predict_examples
+
+
+if __name__ == "__main__":
+  flags.mark_flag_as_required("data_dir")
+  flags.mark_flag_as_required("task_name")
+  flags.mark_flag_as_required("vocab_file")
+  flags.mark_flag_as_required("bert_config_file")
+  flags.mark_flag_as_required("output_dir")
+  tf.compat.v1.app.run()
\ No newline at end of file
diff --git a/models/language_modeling/tensorflow/bert_large/inference/gpu/run_squad.py b/models/language_modeling/tensorflow/bert_large/inference/gpu/run_squad.py
new file mode 100755
index 000000000..25e4ed92d
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/gpu/run_squad.py
@@ -0,0 +1,1497 @@
+#! /usr/bin/env python
+
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run BERT on SQuAD 1.1 and SQuAD 2.0."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import json
+import math
+import os
+import random
+import modeling
+import optimization
+import tokenization
+import six
+import tensorflow as tf
+import generic_ops as bf
+import time
+from datetime import datetime
+
+import sys
+from tensorflow.core.protobuf import config_pb2
+
+from tensorflow.core.protobuf import rewriter_config_pb2
+from tensorflow.python.training import training_util
+from tensorflow.python.training import session_run_hook
+from tensorflow.python.platform import gfile
+from tensorflow.python.client import timeline
+from tensorflow.python.ops import state_ops
+from tensorflow.python.training.basic_session_run_hooks import StopAtStepHook, ProfilerHook
+from tensorflow.python import ops
+from absl import app
+#from absl import flags
+from absl import logging
+
+class LoggerHook(tf.estimator.SessionRunHook):
+  """ Logs runtime. """
+  def __init__(self, batch_size, run_profile):
+    self.batch_size = batch_size
+    self.run_profile = run_profile
+
+  def begin(self):
+    self._step = 0
+    self._total_duration = 0
+    self._warmup = 2
+    self._global_step_tensor = training_util._get_or_create_global_step_read()
+
+  def before_run(self, run_context):
+    self._start_time = datetime.now()
+    opts = config_pb2.RunOptions(trace_level=config_pb2.RunOptions.FULL_TRACE)
+    requests = {"global_step": self._global_step_tensor}
+    if self.run_profile:
+      return session_run_hook.SessionRunArgs(requests, options=opts)
+    else:
+      return session_run_hook.SessionRunArgs(requests)
+
+  def after_run(self, run_context, run_values):
+    self._step += 1
+    duration = datetime.now() - self._start_time
+    ms = duration.total_seconds() * 1000.00
+    if self._step > self._warmup:
+      self._total_duration += ms
+      if self._step % 1 == 0:
+        print("Current step: %d, time in ms: %.2f" %(self._step, ms))
+    else:
+      print("Warmup step: %d, time in ms: %.2f" %(self._step, ms))
+    sys.stdout.flush()
+    if self._step == 4 and self.run_profile:
+      with gfile.Open('timeline-bert.json', "w") as f:
+        trace = timeline.Timeline(run_values.run_metadata.step_stats)
+        f.write(trace.generate_chrome_trace_format())
+
+  def end(self, run_context):
+    print("self._step: %d" %self._step)
+    print("Total time spent (after warmup): %.2f ms" %(self._total_duration))
+    print("Time spent per iteration (after warmup): %.2f ms" %(self._total_duration/(self._step - self._warmup)))
+    time_takes = self._total_duration / (self._step - self._warmup)
+    if self.batch_size == 1:
+      print("Latency is %.3f ms" % (time_takes))
+    print("Throughput is %.3f samples/sec" % (self.batch_size * 1000 / time_takes))
+    sys.stdout.flush()
+
+flags = tf.compat.v1.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+    "bert_config_file", None,
+    "The config json file corresponding to the pre-trained BERT model. "
+    "This specifies the model architecture.")
+
+flags.DEFINE_string("vocab_file", None,
+                    "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_string(
+    "output_dir", None,
+    "The output directory where the model checkpoints will be written.")
+
+flags.DEFINE_string(
+    "mode", 'benchmark',
+    "One of three options: 'benchmark'/'profile'/'accuracy'.")
+
+## Other parameters
+flags.DEFINE_string("train_file", None,
+                    "SQuAD json for training. E.g., train-v1.1.json")
+
+flags.DEFINE_string(
+    "predict_file", None,
+    "SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json")
+
+flags.DEFINE_string(
+    "init_checkpoint", None,
+    "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_bool(
+    "do_lower_case", True,
+    "Whether to lower case the input text. Should be True for uncased "
+    "models and False for cased models.")
+
+flags.DEFINE_integer(
+    "max_seq_length", 384,
+    "The maximum total input sequence length after WordPiece tokenization. "
+    "Sequences longer than this will be truncated, and sequences shorter "
+    "than this will be padded.")
+
+flags.DEFINE_integer(
+    "doc_stride", 128,
+    "When splitting up a long document into chunks, how much stride to "
+    "take between chunks.")
+
+flags.DEFINE_integer(
+    "max_query_length", 64,
+    "The maximum number of tokens for the question. Questions longer than "
+    "this will be truncated to this length.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_predict", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("accum_steps", 1, "Accumulation steps for batch size greater than 32.")
+
+flags.DEFINE_integer("predict_batch_size", 8,
+                     "Total batch size for predictions.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_float("num_train_epochs", 3.0,
+                   "Total number of training epochs to perform.")
+
+flags.DEFINE_float(
+    "warmup_proportion", 0.1,
+    "Proportion of training to perform linear learning rate warmup for. "
+    "E.g., 0.1 = 10% of training.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+                     "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+                     "How many steps to make in each estimator call.")
+
+flags.DEFINE_integer(
+    "n_best_size", 20,
+    "The total number of n-best predictions to generate in the "
+    "nbest_predictions.json output file.")
+
+flags.DEFINE_integer(
+    "max_answer_length", 30,
+    "The maximum length of an answer that can be generated. This is needed "
+    "because the start and end predictions are not conditioned on one another.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+flags.DEFINE_string(
+    "tpu_name", None,
+    "The Cloud TPU to use for training. This should be either the name "
+    "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+    "url.")
+
+flags.DEFINE_string(
+    "tpu_zone", None,
+    "[Optional] GCE zone where the Cloud TPU is located in. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string(
+    "gcp_project", None,
+    "[Optional] Project name for the Cloud TPU-enabled project. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+flags.DEFINE_string("precision", "fp32", "[Optional] TensorFlow training precision.")
+
+flags.DEFINE_integer(
+    "num_tpu_cores", 8,
+    "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+flags.DEFINE_bool(
+    "verbose_logging", False,
+    "If true, all of the warnings related to data processing will be printed. "
+    "A number of warnings are expected for a normal SQuAD evaluation.")
+
+flags.DEFINE_bool(
+    "version_2_with_negative", False,
+    "If true, the SQuAD examples contain some that do not have an answer.")
+
+flags.DEFINE_float(
+    "null_score_diff_threshold", 0.0,
+    "If null_score - best_non_null is greater than the threshold predict null.")
+
+flags.DEFINE_integer( "inter_op_parallelism_threads", 1,
+                      "Setting inter op for the model")
+
+flags.DEFINE_integer("intra_op_parallelism_threads", 28,
+                     "Setting inter op for the model")
+
+flags.DEFINE_bool("profile", False, "[Optional] To enable Tensorflow profile hook."
+                                    "The profile output will be generated in the output_dir")
+
+flags.DEFINE_string("input_graph", None, "path of frozen graph.")
+
+flags.DEFINE_bool("experimental_gelu", False,
+    "[Optional] If true, use experimental gelu op in model."
+    "           Be careful this flag will crash model with incompatible TF.")
+
+flags.DEFINE_bool("optimized_softmax", False,
+    "[Optional] If true, use optimized softmax op in model.")
+
+flags.DEFINE_integer("warmup_steps", 10,
+    "[Optional] Number of warmup steps.")
+
+flags.DEFINE_integer("steps", 30,
+    "[Optional] Number of benchmark steps.")
+
+
+class UpdateGlobalStepHook(session_run_hook.SessionRunHook):
+  def __init__(self):
+    pass
+
+  def begin(self):
+    self._global_step_tensor = training_util.get_global_step()
+    if self._global_step_tensor is None:
+      raise RuntimeError("Global step should be created to use UpdateGlobalStepHook.")
+    tf.compat.v1.get_default_graph()._unsafe_unfinalize()
+    self._updated_global_step = state_ops.assign_add(self._global_step_tensor, 1, use_locking=True)
+  def after_create_session(self, session, coord):
+    pass
+
+  def before_run(self, run_context):
+    return session_run_hook.SessionRunArgs(self._updated_global_step)
+
+  def after_run(self, run_context, run_values):
+    pass
+
+class SquadExample(object):
+  """A single training/test example for simple sequence classification.
+
+     For examples without an answer, the start and end position are -1.
+  """
+
+  def __init__(self,
+               qas_id,
+               question_text,
+               doc_tokens,
+               orig_answer_text=None,
+               start_position=None,
+               end_position=None,
+               is_impossible=False):
+    self.qas_id = qas_id
+    self.question_text = question_text
+    self.doc_tokens = doc_tokens
+    self.orig_answer_text = orig_answer_text
+    self.start_position = start_position
+    self.end_position = end_position
+    self.is_impossible = is_impossible
+
+  def __str__(self):
+    return self.__repr__()
+
+  def __repr__(self):
+    s = ""
+    s += "qas_id: %s" % (tokenization.printable_text(self.qas_id))
+    s += ", question_text: %s" % (
+        tokenization.printable_text(self.question_text))
+    s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
+    if self.start_position:
+      s += ", start_position: %d" % (self.start_position)
+    if self.start_position:
+      s += ", end_position: %d" % (self.end_position)
+    if self.start_position:
+      s += ", is_impossible: %r" % (self.is_impossible)
+    return s
+
+
+class InputFeatures(object):
+  """A single set of features of data."""
+
+  def __init__(self,
+               unique_id,
+               example_index,
+               doc_span_index,
+               tokens,
+               token_to_orig_map,
+               token_is_max_context,
+               input_ids,
+               input_mask,
+               segment_ids,
+               start_position=None,
+               end_position=None,
+               is_impossible=None):
+    self.unique_id = unique_id
+    self.example_index = example_index
+    self.doc_span_index = doc_span_index
+    self.tokens = tokens
+    self.token_to_orig_map = token_to_orig_map
+    self.token_is_max_context = token_is_max_context
+    self.input_ids = input_ids
+    self.input_mask = input_mask
+    self.segment_ids = segment_ids
+    self.start_position = start_position
+    self.end_position = end_position
+    self.is_impossible = is_impossible
+
+
+def read_squad_examples(input_file, is_training):
+  """Read a SQuAD json file into a list of SquadExample."""
+  with tf.io.gfile.GFile(input_file, "r") as reader:
+    input_data = json.load(reader)["data"]
+
+  def is_whitespace(c):
+    if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
+      return True
+    return False
+
+  examples = []
+  for entry in input_data:
+    for paragraph in entry["paragraphs"]:
+      paragraph_text = paragraph["context"]
+      doc_tokens = []
+      char_to_word_offset = []
+      prev_is_whitespace = True
+      for c in paragraph_text:
+        if is_whitespace(c):
+          prev_is_whitespace = True
+        else:
+          if prev_is_whitespace:
+            doc_tokens.append(c)
+          else:
+            doc_tokens[-1] += c
+          prev_is_whitespace = False
+        char_to_word_offset.append(len(doc_tokens) - 1)
+
+      for qa in paragraph["qas"]:
+        qas_id = qa["id"]
+        question_text = qa["question"]
+        start_position = None
+        end_position = None
+        orig_answer_text = None
+        is_impossible = False
+        if is_training:
+
+          if FLAGS.version_2_with_negative:
+            is_impossible = qa["is_impossible"]
+          if (len(qa["answers"]) != 1) and (not is_impossible):
+            raise ValueError(
+                "For training, each question should have exactly 1 answer.")
+          if not is_impossible:
+            answer = qa["answers"][0]
+            orig_answer_text = answer["text"]
+            answer_offset = answer["answer_start"]
+            answer_length = len(orig_answer_text)
+            start_position = char_to_word_offset[answer_offset]
+            end_position = char_to_word_offset[answer_offset + answer_length -
+                                               1]
+            # Only add answers where the text can be exactly recovered from the
+            # document. If this CAN'T happen it's likely due to weird Unicode
+            # stuff so we will just skip the example.
+            #
+            # Note that this means for training mode, every example is NOT
+            # guaranteed to be preserved.
+            actual_text = " ".join(
+                doc_tokens[start_position:(end_position + 1)])
+            cleaned_answer_text = " ".join(
+                tokenization.whitespace_tokenize(orig_answer_text))
+            if actual_text.find(cleaned_answer_text) == -1:
+              tf.compat.v1.logging.warning("Could not find answer: '%s' vs. '%s'",
+                                 actual_text, cleaned_answer_text)
+              continue
+          else:
+            start_position = -1
+            end_position = -1
+            orig_answer_text = ""
+
+        example = SquadExample(
+            qas_id=qas_id,
+            question_text=question_text,
+            doc_tokens=doc_tokens,
+            orig_answer_text=orig_answer_text,
+            start_position=start_position,
+            end_position=end_position,
+            is_impossible=is_impossible)
+        examples.append(example)
+
+  return examples
+
+
+def convert_examples_to_features(examples, tokenizer, max_seq_length,
+                                 doc_stride, max_query_length, is_training,
+                                 output_fn):
+  """Loads a data file into a list of `InputBatch`s."""
+
+  unique_id = 1000000000
+
+  for (example_index, example) in enumerate(examples):
+    query_tokens = tokenizer.tokenize(example.question_text)
+
+    if len(query_tokens) > max_query_length:
+      query_tokens = query_tokens[0:max_query_length]
+
+    tok_to_orig_index = []
+    orig_to_tok_index = []
+    all_doc_tokens = []
+    for (i, token) in enumerate(example.doc_tokens):
+      orig_to_tok_index.append(len(all_doc_tokens))
+      sub_tokens = tokenizer.tokenize(token)
+      for sub_token in sub_tokens:
+        tok_to_orig_index.append(i)
+        all_doc_tokens.append(sub_token)
+
+    tok_start_position = None
+    tok_end_position = None
+    if is_training and example.is_impossible:
+      tok_start_position = -1
+      tok_end_position = -1
+    if is_training and not example.is_impossible:
+      tok_start_position = orig_to_tok_index[example.start_position]
+      if example.end_position < len(example.doc_tokens) - 1:
+        tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
+      else:
+        tok_end_position = len(all_doc_tokens) - 1
+      (tok_start_position, tok_end_position) = _improve_answer_span(
+          all_doc_tokens, tok_start_position, tok_end_position, tokenizer,
+          example.orig_answer_text)
+
+    # The -3 accounts for [CLS], [SEP] and [SEP]
+    max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
+
+    # We can have documents that are longer than the maximum sequence length.
+    # To deal with this we do a sliding window approach, where we take chunks
+    # of the up to our max length with a stride of `doc_stride`.
+    _DocSpan = collections.namedtuple(  # pylint: disable=invalid-name
+        "DocSpan", ["start", "length"])
+    doc_spans = []
+    start_offset = 0
+    while start_offset < len(all_doc_tokens):
+      length = len(all_doc_tokens) - start_offset
+      if length > max_tokens_for_doc:
+        length = max_tokens_for_doc
+      doc_spans.append(_DocSpan(start=start_offset, length=length))
+      if start_offset + length == len(all_doc_tokens):
+        break
+      start_offset += min(length, doc_stride)
+
+    for (doc_span_index, doc_span) in enumerate(doc_spans):
+      tokens = []
+      token_to_orig_map = {}
+      token_is_max_context = {}
+      segment_ids = []
+      tokens.append("[CLS]")
+      segment_ids.append(0)
+      for token in query_tokens:
+        tokens.append(token)
+        segment_ids.append(0)
+      tokens.append("[SEP]")
+      segment_ids.append(0)
+
+      for i in range(doc_span.length):
+        split_token_index = doc_span.start + i
+        token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index]
+
+        is_max_context = _check_is_max_context(doc_spans, doc_span_index,
+                                               split_token_index)
+        token_is_max_context[len(tokens)] = is_max_context
+        tokens.append(all_doc_tokens[split_token_index])
+        segment_ids.append(1)
+      tokens.append("[SEP]")
+      segment_ids.append(1)
+
+      input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+      # The mask has 1 for real tokens and 0 for padding tokens. Only real
+      # tokens are attended to.
+      input_mask = [1] * len(input_ids)
+
+      # Zero-pad up to the sequence length.
+      while len(input_ids) < max_seq_length:
+        input_ids.append(0)
+        input_mask.append(0)
+        segment_ids.append(0)
+
+      assert len(input_ids) == max_seq_length
+      assert len(input_mask) == max_seq_length
+      assert len(segment_ids) == max_seq_length
+
+      start_position = None
+      end_position = None
+      if is_training and not example.is_impossible:
+        # For training, if our document chunk does not contain an annotation
+        # we throw it out, since there is nothing to predict.
+        doc_start = doc_span.start
+        doc_end = doc_span.start + doc_span.length - 1
+        out_of_span = False
+        if not (tok_start_position >= doc_start and
+                tok_end_position <= doc_end):
+          out_of_span = True
+        if out_of_span:
+          start_position = 0
+          end_position = 0
+        else:
+          doc_offset = len(query_tokens) + 2
+          start_position = tok_start_position - doc_start + doc_offset
+          end_position = tok_end_position - doc_start + doc_offset
+
+      if is_training and example.is_impossible:
+        start_position = 0
+        end_position = 0
+
+      if example_index < 20:
+        tf.compat.v1.logging.info("*** Example ***")
+        tf.compat.v1.logging.info("unique_id: %s" % (unique_id))
+        tf.compat.v1.logging.info("example_index: %s" % (example_index))
+        tf.compat.v1.logging.info("doc_span_index: %s" % (doc_span_index))
+        tf.compat.v1.logging.info("tokens: %s" % " ".join(
+            [tokenization.printable_text(x) for x in tokens]))
+        tf.compat.v1.logging.info("token_to_orig_map: %s" % " ".join(
+            ["%d:%d" % (x, y) for (x, y) in six.iteritems(token_to_orig_map)]))
+        tf.compat.v1.logging.info("token_is_max_context: %s" % " ".join([
+            "%d:%s" % (x, y) for (x, y) in six.iteritems(token_is_max_context)
+        ]))
+        tf.compat.v1.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+        tf.compat.v1.logging.info(
+            "input_mask: %s" % " ".join([str(x) for x in input_mask]))
+        tf.compat.v1.logging.info(
+            "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+        if is_training and example.is_impossible:
+          tf.compat.v1.logging.info("impossible example")
+        if is_training and not example.is_impossible:
+          answer_text = " ".join(tokens[start_position:(end_position + 1)])
+          tf.compat.v1.logging.info("start_position: %d" % (start_position))
+          tf.compat.v1.logging.info("end_position: %d" % (end_position))
+          tf.compat.v1.logging.info(
+              "answer: %s" % (tokenization.printable_text(answer_text)))
+
+      feature = InputFeatures(
+          unique_id=unique_id,
+          example_index=example_index,
+          doc_span_index=doc_span_index,
+          tokens=tokens,
+          token_to_orig_map=token_to_orig_map,
+          token_is_max_context=token_is_max_context,
+          input_ids=input_ids,
+          input_mask=input_mask,
+          segment_ids=segment_ids,
+          start_position=start_position,
+          end_position=end_position,
+          is_impossible=example.is_impossible)
+
+      # Run callback
+      output_fn(feature)
+
+      unique_id += 1
+
+
+def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer,
+                         orig_answer_text):
+  """Returns tokenized answer spans that better match the annotated answer."""
+
+  # The SQuAD annotations are character based. We first project them to
+  # whitespace-tokenized words. But then after WordPiece tokenization, we can
+  # often find a "better match". For example:
+  #
+  #   Question: What year was John Smith born?
+  #   Context: The leader was John Smith (1895-1943).
+  #   Answer: 1895
+  #
+  # The original whitespace-tokenized answer will be "(1895-1943).". However
+  # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match
+  # the exact answer, 1895.
+  #
+  # However, this is not always possible. Consider the following:
+  #
+  #   Question: What country is the top exporter of electornics?
+  #   Context: The Japanese electronics industry is the lagest in the world.
+  #   Answer: Japan
+  #
+  # In this case, the annotator chose "Japan" as a character sub-span of
+  # the word "Japanese". Since our WordPiece tokenizer does not split
+  # "Japanese", we just use "Japanese" as the annotation. This is fairly rare
+  # in SQuAD, but does happen.
+  tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
+
+  for new_start in range(input_start, input_end + 1):
+    for new_end in range(input_end, new_start - 1, -1):
+      text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
+      if text_span == tok_answer_text:
+        return (new_start, new_end)
+
+  return (input_start, input_end)
+
+
+def _check_is_max_context(doc_spans, cur_span_index, position):
+  """Check if this is the 'max context' doc span for the token."""
+
+  # Because of the sliding window approach taken to scoring documents, a single
+  # token can appear in multiple documents. E.g.
+  #  Doc: the man went to the store and bought a gallon of milk
+  #  Span A: the man went to the
+  #  Span B: to the store and bought
+  #  Span C: and bought a gallon of
+  #  ...
+  #
+  # Now the word 'bought' will have two scores from spans B and C. We only
+  # want to consider the score with "maximum context", which we define as
+  # the *minimum* of its left and right context (the *sum* of left and
+  # right context will always be the same, of course).
+  #
+  # In the example the maximum context for 'bought' would be span C since
+  # it has 1 left context and 3 right context, while span B has 4 left context
+  # and 0 right context.
+  best_score = None
+  best_span_index = None
+  for (span_index, doc_span) in enumerate(doc_spans):
+    end = doc_span.start + doc_span.length - 1
+    if position < doc_span.start:
+      continue
+    if position > end:
+      continue
+    num_left_context = position - doc_span.start
+    num_right_context = end - position
+    score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
+    if best_score is None or score > best_score:
+      best_score = score
+      best_span_index = span_index
+
+  return cur_span_index == best_span_index
+
+
+def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
+                 use_one_hot_embeddings):
+  """Creates a classification model."""
+  model = modeling.BertModel(
+      config=bert_config,
+      is_training=is_training,
+      input_ids=input_ids,
+      input_mask=input_mask,
+      token_type_ids=segment_ids,
+      use_one_hot_embeddings=use_one_hot_embeddings)
+
+  final_hidden = model.get_sequence_output()
+
+  final_hidden_shape = modeling.get_shape_list(final_hidden, expected_rank=3)
+  batch_size = final_hidden_shape[0]
+  seq_length = final_hidden_shape[1]
+  hidden_size = final_hidden_shape[2]
+
+  output_weights = tf.compat.v1.get_variable(
+      "cls/squad/output_weights", [2, hidden_size],
+      initializer=tf.compat.v1.truncated_normal_initializer(stddev=0.02))
+
+  output_bias = tf.compat.v1.get_variable(
+      "cls/squad/output_bias", [2], initializer=tf.compat.v1.zeros_initializer())
+
+  final_hidden_matrix = tf.reshape(final_hidden,
+                                   [batch_size * seq_length, hidden_size])
+  logits = tf.matmul(final_hidden_matrix, output_weights, transpose_b=True)
+  logits = tf.nn.bias_add(logits, output_bias)
+
+  logits = tf.reshape(logits, [batch_size, seq_length, 2])
+  logits = tf.transpose(a=logits, perm=[2, 0, 1])
+
+  unstacked_logits = tf.unstack(logits, axis=0)
+
+  (start_logits, end_logits) = (unstacked_logits[0], unstacked_logits[1])
+  start = tf.identity(start_logits, name = "start_logits")
+  end = tf.identity(end_logits, name = "end_logits")
+
+  return (start, end)
+
+
+def create_model_top(bert_config, is_training, input_ids, input_mask, segment_ids,
+                     use_one_hot_embeddings, frozen_graph=None):
+    if frozen_graph and not is_training:
+        print("reading***** From *** pb", flush=True)
+        inputs ={'input_ids': input_ids,
+                 'input_mask': input_mask,
+                 'segment_ids': segment_ids}
+
+        outputs = ['unstacked_logits:0', 'unstacked_logits:1']
+ 
+        with tf.io.gfile.GFile(frozen_graph, "rb") as f:
+            graph_def = tf.compat.v1.GraphDef()
+            graph_def.ParseFromString(f.read())
+
+        return tf.graph_util.import_graph_def(graph_def, inputs, outputs, name="")
+
+    return create_model(
+        bert_config=bert_config,
+        is_training=is_training,
+        input_ids=input_ids,
+        input_mask=input_mask,
+        segment_ids=segment_ids,
+        use_one_hot_embeddings=use_one_hot_embeddings)
+
+
+def model_fn_builder(bert_config, init_checkpoint, learning_rate,
+                     num_train_steps, num_warmup_steps, use_tpu,
+                     use_one_hot_embeddings, frozen_graph):
+  """Returns `model_fn` closure for TPUEstimator."""
+
+  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
+    """The `model_fn` for TPUEstimator."""
+
+    tf.compat.v1.logging.info("*** Features ***")
+    for name in sorted(features.keys()):
+      tf.compat.v1.logging.info("  name = %s, shape = %s" % (name, features[name].shape))
+
+    unique_ids = features["unique_ids"]
+    input_ids = features["input_ids"]
+    input_mask = features["input_mask"]
+    segment_ids = features["segment_ids"]
+
+    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+    if bert_config.precision == "bfloat16" :
+      with tf.compat.v1.tpu.bfloat16_scope():
+        (start_logits, end_logits) = create_model_top(
+            bert_config=bert_config,
+            is_training=is_training,
+            input_ids=input_ids,
+            input_mask=input_mask,
+            segment_ids=segment_ids,
+            use_one_hot_embeddings=use_one_hot_embeddings,
+            frozen_graph=frozen_graph)
+      start_logits = tf.cast(start_logits, tf.float32)
+      end_logits = tf.cast(end_logits, tf.float32)
+    else :
+        (start_logits, end_logits) = create_model_top(
+            bert_config=bert_config,
+            is_training=is_training,
+            input_ids=input_ids,
+            input_mask=input_mask,
+            segment_ids=segment_ids,
+            use_one_hot_embeddings=use_one_hot_embeddings,
+            frozen_graph=frozen_graph)
+
+    tvars = tf.compat.v1.trainable_variables()
+
+    initialized_variable_names = {}
+    scaffold_fn = None
+    if init_checkpoint:
+      (assignment_map, initialized_variable_names
+      ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint, "SQuAD")
+      if use_tpu:
+
+        def tpu_scaffold():
+          tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+          return tf.compat.v1.train.Scaffold()
+
+        scaffold_fn = tpu_scaffold
+      else:
+        tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+    tf.compat.v1.logging.info("**** Trainable Variables ****")
+    for var in tvars:
+      init_string = ""
+      if var.name in initialized_variable_names:
+        init_string = ", *INIT_FROM_CKPT*"
+      tf.compat.v1.logging.info("  name = %s, shape = %s%s", var.name, var.shape,
+                      init_string)
+
+    output_spec = None
+    if mode == tf.estimator.ModeKeys.TRAIN:
+      seq_length = modeling.get_shape_list(input_ids)[1]
+
+      def compute_loss(logits, positions):
+        one_hot_positions = tf.one_hot(
+            positions, depth=seq_length, dtype=tf.float32)
+        log_probs = tf.nn.log_softmax(logits, axis=-1)
+        loss = -tf.reduce_mean(
+            input_tensor=tf.reduce_sum(input_tensor=one_hot_positions * log_probs, axis=-1))
+        return loss
+
+      start_positions = features["start_positions"]
+      end_positions = features["end_positions"]
+
+      start_loss = compute_loss(start_logits, start_positions)
+      end_loss = compute_loss(end_logits, end_positions)
+
+      total_loss = (start_loss + end_loss) / 2.0
+
+      train_op = optimization.create_optimizer(
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu)
+
+      log_hook = bf.logTheLossHook(total_loss, n=3)
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode,
+          loss=total_loss,
+          train_op=train_op,
+          training_hooks=[log_hook],
+          scaffold_fn=scaffold_fn)
+    elif mode == tf.estimator.ModeKeys.PREDICT:
+      predictions = {
+          "unique_ids": unique_ids,
+          "start_logits": start_logits,
+          "end_logits": end_logits,
+      }
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)
+    else:
+      raise ValueError(
+          "Only TRAIN and PREDICT modes are supported: %s" % (mode))
+
+    return output_spec
+
+  return model_fn
+
+
+def input_fn_builder(input_file, seq_length, batch_size, is_training, drop_remainder):
+  """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+  name_to_features = {
+      "unique_ids": tf.io.FixedLenFeature([], tf.int64),
+      "input_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "input_mask": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "segment_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+  }
+
+  if is_training:
+    name_to_features["start_positions"] = tf.io.FixedLenFeature([], tf.int64)
+    name_to_features["end_positions"] = tf.io.FixedLenFeature([], tf.int64)
+
+  def _decode_record(record, name_to_features):
+    """Decodes a record to a TensorFlow example."""
+    example = tf.io.parse_single_example(serialized=record, features=name_to_features)
+
+    # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+    # So cast all int64 to int32.
+    for name in list(example.keys()):
+      t = example[name]
+      if t.dtype == tf.int64:
+        t = tf.cast(t, dtype=tf.int32)
+      example[name] = t
+
+    return example
+
+  def input_fn(params):
+    """The actual input function."""
+
+    # For training, we want a lot of parallel reading and shuffling.
+    # For eval, we want no shuffling and parallel reading doesn't matter.
+    d = tf.data.TFRecordDataset(input_file)
+    if is_training:
+      d = d.repeat()
+      d = d.shuffle(buffer_size=100)
+
+    d = d.apply(
+        tf.data.experimental.map_and_batch(
+            lambda record: _decode_record(record, name_to_features),
+            batch_size=batch_size,
+            drop_remainder=drop_remainder))
+
+    return d
+
+  return input_fn
+
+
+RawResult = collections.namedtuple("RawResult",
+                                   ["unique_id", "start_logits", "end_logits"])
+
+
+def write_predictions(all_examples, all_features, all_results, n_best_size,
+                      max_answer_length, do_lower_case, output_prediction_file,
+                      output_nbest_file, output_null_log_odds_file):
+  """Write final predictions to the json file and log-odds of null if needed."""
+  tf.compat.v1.logging.info("Writing predictions to: %s" % (output_prediction_file))
+  tf.compat.v1.logging.info("Writing nbest to: %s" % (output_nbest_file))
+
+  example_index_to_features = collections.defaultdict(list)
+  for feature in all_features:
+    example_index_to_features[feature.example_index].append(feature)
+
+  unique_id_to_result = {}
+  for result in all_results:
+    unique_id_to_result[result.unique_id] = result
+
+  _PrelimPrediction = collections.namedtuple(  # pylint: disable=invalid-name
+      "PrelimPrediction",
+      ["feature_index", "start_index", "end_index", "start_logit", "end_logit"])
+
+  all_predictions = collections.OrderedDict()
+  all_nbest_json = collections.OrderedDict()
+  scores_diff_json = collections.OrderedDict()
+
+  for (example_index, example) in enumerate(all_examples):
+    features = example_index_to_features[example_index]
+
+    prelim_predictions = []
+    # keep track of the minimum score of null start+end of position 0
+    score_null = 1000000  # large and positive
+    min_null_feature_index = 0  # the paragraph slice with min mull score
+    null_start_logit = 0  # the start logit at the slice with min null score
+    null_end_logit = 0  # the end logit at the slice with min null score
+    for (feature_index, feature) in enumerate(features):
+      result = unique_id_to_result[feature.unique_id]
+      start_indexes = _get_best_indexes(result.start_logits, n_best_size)
+      end_indexes = _get_best_indexes(result.end_logits, n_best_size)
+      # if we could have irrelevant answers, get the min score of irrelevant
+      if FLAGS.version_2_with_negative:
+        feature_null_score = result.start_logits[0] + result.end_logits[0]
+        if feature_null_score < score_null:
+          score_null = feature_null_score
+          min_null_feature_index = feature_index
+          null_start_logit = result.start_logits[0]
+          null_end_logit = result.end_logits[0]
+      for start_index in start_indexes:
+        for end_index in end_indexes:
+          # We could hypothetically create invalid predictions, e.g., predict
+          # that the start of the span is in the question. We throw out all
+          # invalid predictions.
+          if start_index >= len(feature.tokens):
+            continue
+          if end_index >= len(feature.tokens):
+            continue
+          if start_index not in feature.token_to_orig_map:
+            continue
+          if end_index not in feature.token_to_orig_map:
+            continue
+          if not feature.token_is_max_context.get(start_index, False):
+            continue
+          if end_index < start_index:
+            continue
+          length = end_index - start_index + 1
+          if length > max_answer_length:
+            continue
+          prelim_predictions.append(
+              _PrelimPrediction(
+                  feature_index=feature_index,
+                  start_index=start_index,
+                  end_index=end_index,
+                  start_logit=result.start_logits[start_index],
+                  end_logit=result.end_logits[end_index]))
+
+    if FLAGS.version_2_with_negative:
+      prelim_predictions.append(
+          _PrelimPrediction(
+              feature_index=min_null_feature_index,
+              start_index=0,
+              end_index=0,
+              start_logit=null_start_logit,
+              end_logit=null_end_logit))
+    prelim_predictions = sorted(
+        prelim_predictions,
+        key=lambda x: (x.start_logit + x.end_logit),
+        reverse=True)
+
+    _NbestPrediction = collections.namedtuple(  # pylint: disable=invalid-name
+        "NbestPrediction", ["text", "start_logit", "end_logit"])
+
+    seen_predictions = {}
+    nbest = []
+    for pred in prelim_predictions:
+      if len(nbest) >= n_best_size:
+        break
+      feature = features[pred.feature_index]
+      if pred.start_index > 0:  # this is a non-null prediction
+        tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
+        orig_doc_start = feature.token_to_orig_map[pred.start_index]
+        orig_doc_end = feature.token_to_orig_map[pred.end_index]
+        orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
+        tok_text = " ".join(tok_tokens)
+
+        # De-tokenize WordPieces that have been split off.
+        tok_text = tok_text.replace(" ##", "")
+        tok_text = tok_text.replace("##", "")
+
+        # Clean whitespace
+        tok_text = tok_text.strip()
+        tok_text = " ".join(tok_text.split())
+        orig_text = " ".join(orig_tokens)
+
+        final_text = get_final_text(tok_text, orig_text, do_lower_case)
+        if final_text in seen_predictions:
+          continue
+
+        seen_predictions[final_text] = True
+      else:
+        final_text = ""
+        seen_predictions[final_text] = True
+
+      nbest.append(
+          _NbestPrediction(
+              text=final_text,
+              start_logit=pred.start_logit,
+              end_logit=pred.end_logit))
+
+    # if we didn't inlude the empty option in the n-best, inlcude it
+    if FLAGS.version_2_with_negative:
+      if "" not in seen_predictions:
+        nbest.append(
+            _NbestPrediction(
+                text="", start_logit=null_start_logit,
+                end_logit=null_end_logit))
+    # In very rare edge cases we could have no valid predictions. So we
+    # just create a nonce prediction in this case to avoid failure.
+    if not nbest:
+      nbest.append(
+          _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
+
+    assert len(nbest) >= 1
+
+    total_scores = []
+    best_non_null_entry = None
+    for entry in nbest:
+      total_scores.append(entry.start_logit + entry.end_logit)
+      if not best_non_null_entry:
+        if entry.text:
+          best_non_null_entry = entry
+
+    probs = _compute_softmax(total_scores)
+
+    nbest_json = []
+    for (i, entry) in enumerate(nbest):
+      output = collections.OrderedDict()
+      output["text"] = entry.text
+      output["probability"] = probs[i]
+      output["start_logit"] = entry.start_logit
+      output["end_logit"] = entry.end_logit
+      nbest_json.append(output)
+
+    assert len(nbest_json) >= 1
+
+    if not FLAGS.version_2_with_negative:
+      all_predictions[example.qas_id] = nbest_json[0]["text"]
+    else:
+      # predict "" iff the null score - the score of best non-null > threshold
+      score_diff = score_null - best_non_null_entry.start_logit - (
+          best_non_null_entry.end_logit)
+      scores_diff_json[example.qas_id] = score_diff
+      if score_diff > FLAGS.null_score_diff_threshold:
+        all_predictions[example.qas_id] = ""
+      else:
+        all_predictions[example.qas_id] = best_non_null_entry.text
+
+    all_nbest_json[example.qas_id] = nbest_json
+
+  with tf.io.gfile.GFile(output_prediction_file, "w") as writer:
+    writer.write(json.dumps(all_predictions, indent=4) + "\n")
+
+  with tf.io.gfile.GFile(output_nbest_file, "w") as writer:
+    writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
+
+  if FLAGS.version_2_with_negative:
+    with tf.io.gfile.GFile(output_null_log_odds_file, "w") as writer:
+      writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
+
+
+def get_final_text(pred_text, orig_text, do_lower_case):
+  """Project the tokenized prediction back to the original text."""
+
+  # When we created the data, we kept track of the alignment between original
+  # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
+  # now `orig_text` contains the span of our original text corresponding to the
+  # span that we predicted.
+  #
+  # However, `orig_text` may contain extra characters that we don't want in
+  # our prediction.
+  #
+  # For example, let's say:
+  #   pred_text = steve smith
+  #   orig_text = Steve Smith's
+  #
+  # We don't want to return `orig_text` because it contains the extra "'s".
+  #
+  # We don't want to return `pred_text` because it's already been normalized
+  # (the SQuAD eval script also does punctuation stripping/lower casing but
+  # our tokenizer does additional normalization like stripping accent
+  # characters).
+  #
+  # What we really want to return is "Steve Smith".
+  #
+  # Therefore, we have to apply a semi-complicated alignment heruistic between
+  # `pred_text` and `orig_text` to get a character-to-charcter alignment. This
+  # can fail in certain cases in which case we just return `orig_text`.
+
+  def _strip_spaces(text):
+    ns_chars = []
+    ns_to_s_map = collections.OrderedDict()
+    for (i, c) in enumerate(text):
+      if c == " ":
+        continue
+      ns_to_s_map[len(ns_chars)] = i
+      ns_chars.append(c)
+    ns_text = "".join(ns_chars)
+    return (ns_text, ns_to_s_map)
+
+  # We first tokenize `orig_text`, strip whitespace from the result
+  # and `pred_text`, and check if they are the same length. If they are
+  # NOT the same length, the heuristic has failed. If they are the same
+  # length, we assume the characters are one-to-one aligned.
+  tokenizer = tokenization.BasicTokenizer(do_lower_case=do_lower_case)
+
+  tok_text = " ".join(tokenizer.tokenize(orig_text))
+
+  start_position = tok_text.find(pred_text)
+  if start_position == -1:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info(
+          "Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
+    return orig_text
+  end_position = start_position + len(pred_text) - 1
+
+  (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
+  (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
+
+  if len(orig_ns_text) != len(tok_ns_text):
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Length not equal after stripping spaces: '%s' vs '%s'",
+                      orig_ns_text, tok_ns_text)
+    return orig_text
+
+  # We then project the characters in `pred_text` back to `orig_text` using
+  # the character-to-character alignment.
+  tok_s_to_ns_map = {}
+  for (i, tok_index) in six.iteritems(tok_ns_to_s_map):
+    tok_s_to_ns_map[tok_index] = i
+
+  orig_start_position = None
+  if start_position in tok_s_to_ns_map:
+    ns_start_position = tok_s_to_ns_map[start_position]
+    if ns_start_position in orig_ns_to_s_map:
+      orig_start_position = orig_ns_to_s_map[ns_start_position]
+
+  if orig_start_position is None:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Couldn't map start position")
+    return orig_text
+
+  orig_end_position = None
+  if end_position in tok_s_to_ns_map:
+    ns_end_position = tok_s_to_ns_map[end_position]
+    if ns_end_position in orig_ns_to_s_map:
+      orig_end_position = orig_ns_to_s_map[ns_end_position]
+
+  if orig_end_position is None:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Couldn't map end position")
+    return orig_text
+
+  output_text = orig_text[orig_start_position:(orig_end_position + 1)]
+  return output_text
+
+
+def _get_best_indexes(logits, n_best_size):
+  """Get the n-best logits from a list."""
+  index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)
+
+  best_indexes = []
+  for i in range(len(index_and_score)):
+    if i >= n_best_size:
+      break
+    best_indexes.append(index_and_score[i][0])
+  return best_indexes
+
+
+def _compute_softmax(scores):
+  """Compute softmax probability over raw logits."""
+  if not scores:
+    return []
+
+  max_score = None
+  for score in scores:
+    if max_score is None or score > max_score:
+      max_score = score
+
+  exp_scores = []
+  total_sum = 0.0
+  for score in scores:
+    x = math.exp(score - max_score)
+    exp_scores.append(x)
+    total_sum += x
+
+  probs = []
+  for score in exp_scores:
+    probs.append(score / total_sum)
+  return probs
+
+
+class FeatureWriter(object):
+  """Writes InputFeature to TF example file."""
+
+  def __init__(self, filename, is_training):
+    self.filename = filename
+    self.is_training = is_training
+    self.num_features = 0
+    self._writer = tf.io.TFRecordWriter(filename)
+
+  def process_feature(self, feature):
+    """Write a InputFeature to the TFRecordWriter as a tf.train.Example."""
+    self.num_features += 1
+
+    def create_int_feature(values):
+      feature = tf.train.Feature(
+          int64_list=tf.train.Int64List(value=list(values)))
+      return feature
+
+    features = collections.OrderedDict()
+    features["unique_ids"] = create_int_feature([feature.unique_id])
+    features["input_ids"] = create_int_feature(feature.input_ids)
+    features["input_mask"] = create_int_feature(feature.input_mask)
+    features["segment_ids"] = create_int_feature(feature.segment_ids)
+
+    if self.is_training:
+      features["start_positions"] = create_int_feature([feature.start_position])
+      features["end_positions"] = create_int_feature([feature.end_position])
+      impossible = 0
+      if feature.is_impossible:
+        impossible = 1
+      features["is_impossible"] = create_int_feature([impossible])
+
+    tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+    self._writer.write(tf_example.SerializeToString())
+
+  def close(self):
+    self._writer.close()
+
+
+def validate_flags_or_throw(bert_config):
+  """Validate the input FLAGS or throw an exception."""
+  tokenization.validate_case_matches_checkpoint(FLAGS.do_lower_case,
+                                                FLAGS.init_checkpoint)
+
+  if not FLAGS.do_train and not FLAGS.do_predict:
+    raise ValueError("At least one of `do_train` or `do_predict` must be True.")
+
+  if FLAGS.do_train:
+    if not FLAGS.train_file:
+      raise ValueError(
+          "If `do_train` is True, then `train_file` must be specified.")
+  if FLAGS.do_predict:
+    if not FLAGS.predict_file:
+      raise ValueError(
+          "If `do_predict` is True, then `predict_file` must be specified.")
+
+  if FLAGS.precision:
+    bert_config.precision = FLAGS.precision
+
+  if FLAGS.max_seq_length > bert_config.max_position_embeddings:
+    raise ValueError(
+        "Cannot use sequence length %d because the BERT model "
+        "was only trained up to sequence length %d" %
+        (FLAGS.max_seq_length, bert_config.max_position_embeddings))
+
+  if FLAGS.max_seq_length <= FLAGS.max_query_length + 3:
+    raise ValueError(
+        "The max_seq_length (%d) must be greater than max_query_length "
+        "(%d) + 3" % (FLAGS.max_seq_length, FLAGS.max_query_length))
+
+
+def main(_):
+  tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
+
+  if (FLAGS.accum_steps >1 ):
+    tf.compat.v1.logging.info(" Accum steps not yet supported in SQuAD")
+    exit(0)
+
+  if FLAGS.profile:
+    tf.compat.v1.disable_eager_execution()
+
+  bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+  bert_config.experimental_gelu = FLAGS.experimental_gelu
+  bert_config.optimized_softmax = FLAGS.optimized_softmax
+
+  validate_flags_or_throw(bert_config)
+
+  tf.io.gfile.makedirs(FLAGS.output_dir)
+
+  tokenizer = tokenization.FullTokenizer(
+      vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+  tpu_cluster_resolver = None
+  if FLAGS.use_tpu and FLAGS.tpu_name:
+    tpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
+        FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+  if bert_config.precision == 'bfloat16':
+    #graph_options = tf.compat.v1.GraphOptions(
+    #    rewrite_options=rewriter_config_pb2.RewriterConfig(
+    #        convert_to_bfloat16=rewriter_config_pb2.RewriterConfig.ON))
+    graph_options = None
+  else:
+    graph_options = None
+  session_config = tf.compat.v1.ConfigProto(graph_options=graph_options,
+      inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
+      intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads,
+      allow_soft_placement=True)
+
+  is_per_host = tf.compat.v1.estimator.tpu.InputPipelineConfig.PER_HOST_V2
+  run_config = tf.compat.v1.estimator.tpu.RunConfig(
+      cluster=tpu_cluster_resolver,
+      master=FLAGS.master,
+      model_dir=FLAGS.output_dir,
+      save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+      session_config=session_config,
+      tpu_config=tf.compat.v1.estimator.tpu.TPUConfig(
+          iterations_per_loop=FLAGS.iterations_per_loop,
+          num_shards=FLAGS.num_tpu_cores,
+          per_host_input_for_training=is_per_host))
+
+  train_examples = None
+  num_train_steps = None
+  num_warmup_steps = None
+  if FLAGS.do_train:
+    train_examples = read_squad_examples(
+        input_file=FLAGS.train_file, is_training=True)
+    num_train_steps = int(
+        len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+    num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+    # Pre-shuffle the input to avoid having to make a very large shuffle
+    # buffer in in the `input_fn`.
+    rng = random.Random(12345)
+    rng.shuffle(train_examples)
+
+  model_fn = model_fn_builder(
+      bert_config=bert_config,
+      init_checkpoint=FLAGS.init_checkpoint,
+      learning_rate=FLAGS.learning_rate,
+      num_train_steps=num_train_steps,
+      num_warmup_steps=num_warmup_steps,
+      use_tpu=FLAGS.use_tpu,
+      use_one_hot_embeddings=FLAGS.use_tpu,
+      frozen_graph=FLAGS.input_graph)
+
+  # If TPU is not available, this will fall back to normal Estimator on CPU
+  # or GPU.
+  estimator = tf.compat.v1.estimator.tpu.TPUEstimator(
+      use_tpu=FLAGS.use_tpu,
+      model_fn=model_fn,
+      config=run_config,
+      train_batch_size=FLAGS.train_batch_size,
+      predict_batch_size=FLAGS.predict_batch_size)
+
+  if FLAGS.do_train:
+    # We write to a temporary file to avoid storing very large constant tensors
+    # in memory.
+    train_writer = FeatureWriter(
+        filename=os.path.join(FLAGS.output_dir, "train.tf_record"),
+        is_training=True)
+    convert_examples_to_features(
+        examples=train_examples,
+        tokenizer=tokenizer,
+        max_seq_length=FLAGS.max_seq_length,
+        doc_stride=FLAGS.doc_stride,
+        max_query_length=FLAGS.max_query_length,
+        is_training=True,
+        output_fn=train_writer.process_feature)
+    train_writer.close()
+
+    tf.compat.v1.logging.info("***** Running training *****")
+    tf.compat.v1.logging.info("  Num orig examples = %d", len(train_examples))
+    tf.compat.v1.logging.info("  Num split examples = %d", train_writer.num_features)
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.train_batch_size)
+    tf.compat.v1.logging.info("  Num steps = %d", num_train_steps)
+    del train_examples
+
+    train_input_fn = input_fn_builder(
+        input_file=train_writer.filename,
+        batch_size=FLAGS.predict_batch_size,
+        seq_length=FLAGS.max_seq_length,
+        is_training=True,
+        drop_remainder=True)
+
+    # Added to profile the run
+    if FLAGS.profile ==True :
+      tf.compat.v1.logging.info("***** Running training with profiler*****")
+      hooks = [tf.compat.v1.train.ProfilerHook(save_steps=3, output_dir=FLAGS.output_dir,
+                                               show_memory=True)]
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps, hooks=hooks)
+    else:
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+  if FLAGS.do_predict:
+    eval_examples = read_squad_examples(
+        input_file=FLAGS.predict_file, is_training=False)
+
+    eval_writer = FeatureWriter(
+        filename=os.path.join(FLAGS.output_dir, "eval.tf_record"),
+        is_training=False)
+    eval_features = []
+
+    def append_feature(feature):
+      eval_features.append(feature)
+      eval_writer.process_feature(feature)
+
+    convert_examples_to_features(
+        examples=eval_examples,
+        tokenizer=tokenizer,
+        max_seq_length=FLAGS.max_seq_length,
+        doc_stride=FLAGS.doc_stride,
+        max_query_length=FLAGS.max_query_length,
+        is_training=False,
+        output_fn=append_feature)
+    eval_writer.close()
+
+    tf.compat.v1.logging.info("***** Running predictions *****")
+    tf.compat.v1.logging.info("  Num orig examples = %d", len(eval_examples))
+    tf.compat.v1.logging.info("  Num split examples = %d", len(eval_features))
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.predict_batch_size)
+
+    all_results = []
+
+    predict_input_fn = input_fn_builder(
+        input_file=eval_writer.filename,
+        batch_size=FLAGS.predict_batch_size,
+        seq_length=FLAGS.max_seq_length,
+        is_training=False,
+        drop_remainder=False)
+
+    # If running eval on the TPU, you will need to specify the number of
+    # steps.
+    tf.compat.v1.disable_eager_execution()
+    warmup_steps = FLAGS.warmup_steps
+    total_steps = FLAGS.warmup_steps + FLAGS.steps
+    if FLAGS.mode == 'benchmark':
+      hooks = [UpdateGlobalStepHook(),
+               StopAtStepHook(total_steps),
+               LoggerHook(FLAGS.predict_batch_size, False)]
+    elif FLAGS.mode == 'profile':
+      hooks = [UpdateGlobalStepHook(),
+               StopAtStepHook(total_steps),
+               ProfilerHook(save_steps=10, output_dir=FLAGS.output_dir),
+               LoggerHook(FLAGS.predict_batch_size, False)]
+    else:
+      hooks = []
+
+    all_results = []
+    if FLAGS.mode in ('benchmark', 'profile'):
+      for result in estimator.predict(
+          predict_input_fn, yield_single_examples=False, hooks=hooks):
+        continue
+    elif FLAGS.mode == "accuracy":
+      for result in estimator.predict(
+          predict_input_fn, yield_single_examples=True,
+          hooks=hooks):
+        if len(all_results) % 10 == 0:
+          tf.compat.v1.logging.info("Processing example: %d" % (len(all_results)))
+        unique_id = int(result["unique_ids"])
+        start_logits = [float(x) for x in result["start_logits"].flat]
+        end_logits = [float(x) for x in result["end_logits"].flat]
+        all_results.append(
+            RawResult(
+                unique_id=unique_id,
+                start_logits=start_logits,
+                end_logits=end_logits))
+      output_prediction_file = os.path.join(FLAGS.output_dir, "predictions.json")
+      output_nbest_file = os.path.join(FLAGS.output_dir, "nbest_predictions.json")
+      output_null_log_odds_file = os.path.join(FLAGS.output_dir, "null_odds.json")
+
+      write_predictions(eval_examples, eval_features, all_results,
+                        FLAGS.n_best_size, FLAGS.max_answer_length,
+                        FLAGS.do_lower_case, output_prediction_file,
+                        output_nbest_file, output_null_log_odds_file)
+
+if __name__ == "__main__":
+  flags.mark_flag_as_required("vocab_file")
+  flags.mark_flag_as_required("bert_config_file")
+  flags.mark_flag_as_required("output_dir")
+  tf.compat.v1.app.run()
diff --git a/models/language_modeling/tensorflow/bert_large/inference/run_squad.py b/models/language_modeling/tensorflow/bert_large/inference/run_squad.py
index 65918eb0c..baca125c3 100755
--- a/models/language_modeling/tensorflow/bert_large/inference/run_squad.py
+++ b/models/language_modeling/tensorflow/bert_large/inference/run_squad.py
@@ -202,6 +202,7 @@
                   "Simulate weight sharing across multiple instances.")
 flags.DEFINE_integer("num_cores_per_socket", None,
                      "Number of cores per socket.")
+flags.DEFINE_bool("onednn_graph", False, "Enable OneDNN Graph for ITEX users.")
 
 class UpdateGlobalStepHook(session_run_hook.SessionRunHook):
   def __init__(self):
@@ -665,12 +666,26 @@ def create_model_top(bert_config, is_training, input_ids, input_mask, segment_id
                  'input_mask': input_mask,
                  'segment_ids': segment_ids}
 
-        outputs = ['start_logits:0', 'end_logits:0']
-
         with tf.io.gfile.GFile(frozen_graph, "rb") as f:
             graph_def = tf.compat.v1.GraphDef()
             graph_def.ParseFromString(f.read())
 
+        if FLAGS.onednn_graph:
+          all_node_name = [n.name for n in graph_def.node]
+
+          outputs = []
+
+          candidate_outputs_1 = ['unstacked_logits:0', 'unstacked_logits:1']
+          candidate_outputs_2 = ['start_logits:0', 'end_logits:0']
+
+          if "unstacked_logits" in all_node_name:
+            outputs = candidate_outputs_1
+
+          if "start_logits" in all_node_name:
+            outputs = candidate_outputs_2
+        else:
+          outputs = ['start_logits:0', 'end_logits:0']
+
         return tf.graph_util.import_graph_def(graph_def, inputs, outputs, name="")
 
     return create_model(
@@ -1301,7 +1316,22 @@ def create_feed_dict():
         'segment_ids:0': segment_ids}
 
   def run_model(sess, tid):
-    outputs = ['start_logits:0', 'end_logits:0']
+    if FLAGS.onednn_graph:
+      all_tensor_name = [tensor.name for op in sess.graph.get_operations() for tensor in op.values()]
+
+      outputs = []
+
+      candidate_outputs_1 = ['unstacked_logits:0', 'unstacked_logits:1']
+      candidate_outputs_2 = ['start_logits:0', 'end_logits:0']
+
+      if candidate_outputs_1[0] in all_tensor_name and candidate_outputs_1[1] in all_tensor_name:
+        outputs = candidate_outputs_1
+
+      if candidate_outputs_2[0] in all_tensor_name and candidate_outputs_2[1] in all_tensor_name:
+        outputs = candidate_outputs_2
+    else:
+      outputs = ['start_logits:0', 'end_logits:0']
+
     feed_dict = create_feed_dict()
     for step in range(FLAGS.warmup_steps):
       _ = sess.run(outputs, feed_dict=feed_dict)
diff --git a/models/language_modeling/tensorflow/bert_large/inference/run_squad_int8.py b/models/language_modeling/tensorflow/bert_large/inference/run_squad_int8.py
new file mode 100755
index 000000000..2a6aa4ee2
--- /dev/null
+++ b/models/language_modeling/tensorflow/bert_large/inference/run_squad_int8.py
@@ -0,0 +1,1615 @@
+#! /usr/bin/env python
+
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run BERT on SQuAD 1.1 and SQuAD 2.0."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import json
+import math
+import os
+import random
+import modeling
+import optimization
+import tokenization
+import six
+import tensorflow as tf
+import generic_ops as bf
+import time
+
+from tensorflow.core.protobuf import rewriter_config_pb2
+from tensorflow.python.training import training_util
+from tensorflow.python.training import session_run_hook
+from tensorflow.python.ops import state_ops
+from tensorflow.python.training.basic_session_run_hooks import StopAtStepHook, ProfilerHook
+from tensorflow.python import ops
+from absl import app
+#from absl import flags
+from absl import logging
+
+tf.config.optimizer.set_experimental_options({"constant_folding": False})
+
+# This script is only executed when the onednn graph option is on.
+import intel_extension_for_tensorflow as itex
+auto_mixed_precision_options = itex.AutoMixedPrecisionOptions()
+auto_mixed_precision_options.data_type = itex.BFLOAT16
+
+auto_mixed_precision_options.allowlist_add= "MatMul,BiasAdd,BatchMatMulV2,Mul,AddV2,Softmax,LayerNorm,Gelu"
+auto_mixed_precision_options.inferlist_remove = "MatMul,BiasAdd,BatchMatMulV2,Mul,AddV2,Softmax,LayerNorm,Gelu"
+auto_mixed_precision_options.denylist_add = "QuantizeV2,Dequantize"
+
+graph_options = itex.GraphOptions(auto_mixed_precision_options=auto_mixed_precision_options)
+graph_options.auto_mixed_precision = itex.ON
+
+config = itex.ConfigProto(graph_options=graph_options)
+try:
+  itex.set_backend("cpu", config)
+except Exception:
+  itex.set_config(config)
+
+flags = tf.compat.v1.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+    "bert_config_file", None,
+    "The config json file corresponding to the pre-trained BERT model. "
+    "This specifies the model architecture.")
+
+flags.DEFINE_string("vocab_file", None,
+                    "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_string(
+    "output_dir", None,
+    "The output directory where the model checkpoints will be written.")
+
+flags.DEFINE_string(
+    "mode", 'benchmark',
+    "One of three options: 'benchmark'/'profile'/'accuracy'.")
+
+## Other parameters
+flags.DEFINE_string("train_file", None,
+                    "SQuAD json for training. E.g., train-v1.1.json")
+
+flags.DEFINE_string(
+    "predict_file", None,
+    "SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json")
+
+flags.DEFINE_string(
+    "init_checkpoint", None,
+    "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_bool(
+    "do_lower_case", True,
+    "Whether to lower case the input text. Should be True for uncased "
+    "models and False for cased models.")
+
+flags.DEFINE_integer(
+    "max_seq_length", 384,
+    "The maximum total input sequence length after WordPiece tokenization. "
+    "Sequences longer than this will be truncated, and sequences shorter "
+    "than this will be padded.")
+
+flags.DEFINE_integer(
+    "doc_stride", 128,
+    "When splitting up a long document into chunks, how much stride to "
+    "take between chunks.")
+
+flags.DEFINE_integer(
+    "max_query_length", 64,
+    "The maximum number of tokens for the question. Questions longer than "
+    "this will be truncated to this length.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_predict", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("accum_steps", 1, "Accumulation steps for batch size greater than 32.")
+
+flags.DEFINE_integer("predict_batch_size", 8,
+                     "Total batch size for predictions.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_float("num_train_epochs", 3.0,
+                   "Total number of training epochs to perform.")
+
+flags.DEFINE_float(
+    "warmup_proportion", 0.1,
+    "Proportion of training to perform linear learning rate warmup for. "
+    "E.g., 0.1 = 10% of training.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+                     "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+                     "How many steps to make in each estimator call.")
+
+flags.DEFINE_integer(
+    "n_best_size", 20,
+    "The total number of n-best predictions to generate in the "
+    "nbest_predictions.json output file.")
+
+flags.DEFINE_integer(
+    "max_answer_length", 30,
+    "The maximum length of an answer that can be generated. This is needed "
+    "because the start and end predictions are not conditioned on one another.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+flags.DEFINE_string(
+    "tpu_name", None,
+    "The Cloud TPU to use for training. This should be either the name "
+    "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+    "url.")
+
+flags.DEFINE_string(
+    "tpu_zone", None,
+    "[Optional] GCE zone where the Cloud TPU is located in. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string(
+    "gcp_project", None,
+    "[Optional] Project name for the Cloud TPU-enabled project. If not "
+    "specified, we will attempt to automatically detect the GCE project from "
+    "metadata.")
+
+flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+flags.DEFINE_string("precision", "fp32", "[Optional] TensorFlow training precision.")
+
+flags.DEFINE_integer(
+    "num_tpu_cores", 8,
+    "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+flags.DEFINE_bool(
+    "verbose_logging", False,
+    "If true, all of the warnings related to data processing will be printed. "
+    "A number of warnings are expected for a normal SQuAD evaluation.")
+
+flags.DEFINE_bool(
+    "version_2_with_negative", False,
+    "If true, the SQuAD examples contain some that do not have an answer.")
+
+flags.DEFINE_float(
+    "null_score_diff_threshold", 0.0,
+    "If null_score - best_non_null is greater than the threshold predict null.")
+
+flags.DEFINE_integer( "inter_op_parallelism_threads", 1,
+                      "Setting inter op for the model")
+
+flags.DEFINE_integer("intra_op_parallelism_threads", 28,
+                     "Setting inter op for the model")
+
+flags.DEFINE_bool("profile", False, "[Optional] To enable Tensorflow profile hook."
+                                    "The profile output will be generated in the output_dir")
+
+flags.DEFINE_string("input_graph", None, "path of frozen graph.")
+
+flags.DEFINE_bool("experimental_gelu", False,
+    "[Optional] If true, use experimental gelu op in model."
+    "           Be careful this flag will crash model with incompatible TF.")
+
+flags.DEFINE_bool("optimized_softmax", False,
+    "[Optional] If true, use optimized softmax op in model.")
+
+flags.DEFINE_integer("warmup_steps", 10,
+    "[Optional] Number of warmup steps.")
+
+flags.DEFINE_integer("steps", 30,
+    "[Optional] Number of benchmark steps.")
+
+flags.DEFINE_bool("weight_sharing", False,
+                  "Simulate weight sharing across multiple instances.")
+flags.DEFINE_integer("num_cores_per_socket", None,
+                     "Number of cores per socket.")
+
+class UpdateGlobalStepHook(session_run_hook.SessionRunHook):
+  def __init__(self):
+    pass
+
+  def begin(self):
+    self._global_step_tensor = training_util.get_global_step()
+    if self._global_step_tensor is None:
+      raise RuntimeError("Global step should be created to use UpdateGlobalStepHook.")
+    tf.compat.v1.get_default_graph()._unsafe_unfinalize()
+    self._updated_global_step = state_ops.assign_add(self._global_step_tensor, 1, use_locking=True)
+  def after_create_session(self, session, coord):
+    pass
+
+  def before_run(self, run_context):
+    return session_run_hook.SessionRunArgs(self._updated_global_step)
+
+  def after_run(self, run_context, run_values):
+    pass
+
+class SquadExample(object):
+  """A single training/test example for simple sequence classification.
+
+     For examples without an answer, the start and end position are -1.
+  """
+
+  def __init__(self,
+               qas_id,
+               question_text,
+               doc_tokens,
+               orig_answer_text=None,
+               start_position=None,
+               end_position=None,
+               is_impossible=False):
+    self.qas_id = qas_id
+    self.question_text = question_text
+    self.doc_tokens = doc_tokens
+    self.orig_answer_text = orig_answer_text
+    self.start_position = start_position
+    self.end_position = end_position
+    self.is_impossible = is_impossible
+
+  def __str__(self):
+    return self.__repr__()
+
+  def __repr__(self):
+    s = ""
+    s += "qas_id: %s" % (tokenization.printable_text(self.qas_id))
+    s += ", question_text: %s" % (
+        tokenization.printable_text(self.question_text))
+    s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
+    if self.start_position:
+      s += ", start_position: %d" % (self.start_position)
+    if self.start_position:
+      s += ", end_position: %d" % (self.end_position)
+    if self.start_position:
+      s += ", is_impossible: %r" % (self.is_impossible)
+    return s
+
+
+class InputFeatures(object):
+  """A single set of features of data."""
+
+  def __init__(self,
+               unique_id,
+               example_index,
+               doc_span_index,
+               tokens,
+               token_to_orig_map,
+               token_is_max_context,
+               input_ids,
+               input_mask,
+               segment_ids,
+               start_position=None,
+               end_position=None,
+               is_impossible=None):
+    self.unique_id = unique_id
+    self.example_index = example_index
+    self.doc_span_index = doc_span_index
+    self.tokens = tokens
+    self.token_to_orig_map = token_to_orig_map
+    self.token_is_max_context = token_is_max_context
+    self.input_ids = input_ids
+    self.input_mask = input_mask
+    self.segment_ids = segment_ids
+    self.start_position = start_position
+    self.end_position = end_position
+    self.is_impossible = is_impossible
+
+
+def read_squad_examples(input_file, is_training):
+  """Read a SQuAD json file into a list of SquadExample."""
+  with tf.io.gfile.GFile(input_file, "r") as reader:
+    input_data = json.load(reader)["data"]
+
+  def is_whitespace(c):
+    if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
+      return True
+    return False
+
+  examples = []
+  for entry in input_data:
+    for paragraph in entry["paragraphs"]:
+      paragraph_text = paragraph["context"]
+      doc_tokens = []
+      char_to_word_offset = []
+      prev_is_whitespace = True
+      for c in paragraph_text:
+        if is_whitespace(c):
+          prev_is_whitespace = True
+        else:
+          if prev_is_whitespace:
+            doc_tokens.append(c)
+          else:
+            doc_tokens[-1] += c
+          prev_is_whitespace = False
+        char_to_word_offset.append(len(doc_tokens) - 1)
+
+      for qa in paragraph["qas"]:
+        qas_id = qa["id"]
+        question_text = qa["question"]
+        start_position = None
+        end_position = None
+        orig_answer_text = None
+        is_impossible = False
+        if is_training:
+
+          if FLAGS.version_2_with_negative:
+            is_impossible = qa["is_impossible"]
+          if (len(qa["answers"]) != 1) and (not is_impossible):
+            raise ValueError(
+                "For training, each question should have exactly 1 answer.")
+          if not is_impossible:
+            answer = qa["answers"][0]
+            orig_answer_text = answer["text"]
+            answer_offset = answer["answer_start"]
+            answer_length = len(orig_answer_text)
+            start_position = char_to_word_offset[answer_offset]
+            end_position = char_to_word_offset[answer_offset + answer_length -
+                                               1]
+            # Only add answers where the text can be exactly recovered from the
+            # document. If this CAN'T happen it's likely due to weird Unicode
+            # stuff so we will just skip the example.
+            #
+            # Note that this means for training mode, every example is NOT
+            # guaranteed to be preserved.
+            actual_text = " ".join(
+                doc_tokens[start_position:(end_position + 1)])
+            cleaned_answer_text = " ".join(
+                tokenization.whitespace_tokenize(orig_answer_text))
+            if actual_text.find(cleaned_answer_text) == -1:
+              tf.compat.v1.logging.warning("Could not find answer: '%s' vs. '%s'",
+                                 actual_text, cleaned_answer_text)
+              continue
+          else:
+            start_position = -1
+            end_position = -1
+            orig_answer_text = ""
+
+        example = SquadExample(
+            qas_id=qas_id,
+            question_text=question_text,
+            doc_tokens=doc_tokens,
+            orig_answer_text=orig_answer_text,
+            start_position=start_position,
+            end_position=end_position,
+            is_impossible=is_impossible)
+        examples.append(example)
+
+  return examples
+
+
+def convert_examples_to_features(examples, tokenizer, max_seq_length,
+                                 doc_stride, max_query_length, is_training,
+                                 output_fn):
+  """Loads a data file into a list of `InputBatch`s."""
+
+  unique_id = 1000000000
+
+  for (example_index, example) in enumerate(examples):
+    query_tokens = tokenizer.tokenize(example.question_text)
+
+    if len(query_tokens) > max_query_length:
+      query_tokens = query_tokens[0:max_query_length]
+
+    tok_to_orig_index = []
+    orig_to_tok_index = []
+    all_doc_tokens = []
+    for (i, token) in enumerate(example.doc_tokens):
+      orig_to_tok_index.append(len(all_doc_tokens))
+      sub_tokens = tokenizer.tokenize(token)
+      for sub_token in sub_tokens:
+        tok_to_orig_index.append(i)
+        all_doc_tokens.append(sub_token)
+
+    tok_start_position = None
+    tok_end_position = None
+    if is_training and example.is_impossible:
+      tok_start_position = -1
+      tok_end_position = -1
+    if is_training and not example.is_impossible:
+      tok_start_position = orig_to_tok_index[example.start_position]
+      if example.end_position < len(example.doc_tokens) - 1:
+        tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
+      else:
+        tok_end_position = len(all_doc_tokens) - 1
+      (tok_start_position, tok_end_position) = _improve_answer_span(
+          all_doc_tokens, tok_start_position, tok_end_position, tokenizer,
+          example.orig_answer_text)
+
+    # The -3 accounts for [CLS], [SEP] and [SEP]
+    max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
+
+    # We can have documents that are longer than the maximum sequence length.
+    # To deal with this we do a sliding window approach, where we take chunks
+    # of the up to our max length with a stride of `doc_stride`.
+    _DocSpan = collections.namedtuple(  # pylint: disable=invalid-name
+        "DocSpan", ["start", "length"])
+    doc_spans = []
+    start_offset = 0
+    while start_offset < len(all_doc_tokens):
+      length = len(all_doc_tokens) - start_offset
+      if length > max_tokens_for_doc:
+        length = max_tokens_for_doc
+      doc_spans.append(_DocSpan(start=start_offset, length=length))
+      if start_offset + length == len(all_doc_tokens):
+        break
+      start_offset += min(length, doc_stride)
+
+    for (doc_span_index, doc_span) in enumerate(doc_spans):
+      tokens = []
+      token_to_orig_map = {}
+      token_is_max_context = {}
+      segment_ids = []
+      tokens.append("[CLS]")
+      segment_ids.append(0)
+      for token in query_tokens:
+        tokens.append(token)
+        segment_ids.append(0)
+      tokens.append("[SEP]")
+      segment_ids.append(0)
+
+      for i in range(doc_span.length):
+        split_token_index = doc_span.start + i
+        token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index]
+
+        is_max_context = _check_is_max_context(doc_spans, doc_span_index,
+                                               split_token_index)
+        token_is_max_context[len(tokens)] = is_max_context
+        tokens.append(all_doc_tokens[split_token_index])
+        segment_ids.append(1)
+      tokens.append("[SEP]")
+      segment_ids.append(1)
+
+      input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+      # The mask has 1 for real tokens and 0 for padding tokens. Only real
+      # tokens are attended to.
+      input_mask = [1] * len(input_ids)
+
+      # Zero-pad up to the sequence length.
+      while len(input_ids) < max_seq_length:
+        input_ids.append(0)
+        input_mask.append(0)
+        segment_ids.append(0)
+
+      assert len(input_ids) == max_seq_length
+      assert len(input_mask) == max_seq_length
+      assert len(segment_ids) == max_seq_length
+
+      start_position = None
+      end_position = None
+      if is_training and not example.is_impossible:
+        # For training, if our document chunk does not contain an annotation
+        # we throw it out, since there is nothing to predict.
+        doc_start = doc_span.start
+        doc_end = doc_span.start + doc_span.length - 1
+        out_of_span = False
+        if not (tok_start_position >= doc_start and
+                tok_end_position <= doc_end):
+          out_of_span = True
+        if out_of_span:
+          start_position = 0
+          end_position = 0
+        else:
+          doc_offset = len(query_tokens) + 2
+          start_position = tok_start_position - doc_start + doc_offset
+          end_position = tok_end_position - doc_start + doc_offset
+
+      if is_training and example.is_impossible:
+        start_position = 0
+        end_position = 0
+
+      if example_index < 20:
+        tf.compat.v1.logging.info("*** Example ***")
+        tf.compat.v1.logging.info("unique_id: %s" % (unique_id))
+        tf.compat.v1.logging.info("example_index: %s" % (example_index))
+        tf.compat.v1.logging.info("doc_span_index: %s" % (doc_span_index))
+        tf.compat.v1.logging.info("tokens: %s" % " ".join(
+            [tokenization.printable_text(x) for x in tokens]))
+        tf.compat.v1.logging.info("token_to_orig_map: %s" % " ".join(
+            ["%d:%d" % (x, y) for (x, y) in six.iteritems(token_to_orig_map)]))
+        tf.compat.v1.logging.info("token_is_max_context: %s" % " ".join([
+            "%d:%s" % (x, y) for (x, y) in six.iteritems(token_is_max_context)
+        ]))
+        tf.compat.v1.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+        tf.compat.v1.logging.info(
+            "input_mask: %s" % " ".join([str(x) for x in input_mask]))
+        tf.compat.v1.logging.info(
+            "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+        if is_training and example.is_impossible:
+          tf.compat.v1.logging.info("impossible example")
+        if is_training and not example.is_impossible:
+          answer_text = " ".join(tokens[start_position:(end_position + 1)])
+          tf.compat.v1.logging.info("start_position: %d" % (start_position))
+          tf.compat.v1.logging.info("end_position: %d" % (end_position))
+          tf.compat.v1.logging.info(
+              "answer: %s" % (tokenization.printable_text(answer_text)))
+
+      feature = InputFeatures(
+          unique_id=unique_id,
+          example_index=example_index,
+          doc_span_index=doc_span_index,
+          tokens=tokens,
+          token_to_orig_map=token_to_orig_map,
+          token_is_max_context=token_is_max_context,
+          input_ids=input_ids,
+          input_mask=input_mask,
+          segment_ids=segment_ids,
+          start_position=start_position,
+          end_position=end_position,
+          is_impossible=example.is_impossible)
+
+      # Run callback
+      output_fn(feature)
+
+      unique_id += 1
+
+
+def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer,
+                         orig_answer_text):
+  """Returns tokenized answer spans that better match the annotated answer."""
+
+  # The SQuAD annotations are character based. We first project them to
+  # whitespace-tokenized words. But then after WordPiece tokenization, we can
+  # often find a "better match". For example:
+  #
+  #   Question: What year was John Smith born?
+  #   Context: The leader was John Smith (1895-1943).
+  #   Answer: 1895
+  #
+  # The original whitespace-tokenized answer will be "(1895-1943).". However
+  # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match
+  # the exact answer, 1895.
+  #
+  # However, this is not always possible. Consider the following:
+  #
+  #   Question: What country is the top exporter of electornics?
+  #   Context: The Japanese electronics industry is the lagest in the world.
+  #   Answer: Japan
+  #
+  # In this case, the annotator chose "Japan" as a character sub-span of
+  # the word "Japanese". Since our WordPiece tokenizer does not split
+  # "Japanese", we just use "Japanese" as the annotation. This is fairly rare
+  # in SQuAD, but does happen.
+  tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
+
+  for new_start in range(input_start, input_end + 1):
+    for new_end in range(input_end, new_start - 1, -1):
+      text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
+      if text_span == tok_answer_text:
+        return (new_start, new_end)
+
+  return (input_start, input_end)
+
+
+def _check_is_max_context(doc_spans, cur_span_index, position):
+  """Check if this is the 'max context' doc span for the token."""
+
+  # Because of the sliding window approach taken to scoring documents, a single
+  # token can appear in multiple documents. E.g.
+  #  Doc: the man went to the store and bought a gallon of milk
+  #  Span A: the man went to the
+  #  Span B: to the store and bought
+  #  Span C: and bought a gallon of
+  #  ...
+  #
+  # Now the word 'bought' will have two scores from spans B and C. We only
+  # want to consider the score with "maximum context", which we define as
+  # the *minimum* of its left and right context (the *sum* of left and
+  # right context will always be the same, of course).
+  #
+  # In the example the maximum context for 'bought' would be span C since
+  # it has 1 left context and 3 right context, while span B has 4 left context
+  # and 0 right context.
+  best_score = None
+  best_span_index = None
+  for (span_index, doc_span) in enumerate(doc_spans):
+    end = doc_span.start + doc_span.length - 1
+    if position < doc_span.start:
+      continue
+    if position > end:
+      continue
+    num_left_context = position - doc_span.start
+    num_right_context = end - position
+    score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
+    if best_score is None or score > best_score:
+      best_score = score
+      best_span_index = span_index
+
+  return cur_span_index == best_span_index
+
+
+def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
+                 use_one_hot_embeddings):
+  """Creates a classification model."""
+  model = modeling.BertModel(
+      config=bert_config,
+      is_training=is_training,
+      input_ids=input_ids,
+      input_mask=input_mask,
+      token_type_ids=segment_ids,
+      use_one_hot_embeddings=use_one_hot_embeddings)
+
+  final_hidden = model.get_sequence_output()
+
+  final_hidden_shape = modeling.get_shape_list(final_hidden, expected_rank=3)
+  batch_size = final_hidden_shape[0]
+  seq_length = final_hidden_shape[1]
+  hidden_size = final_hidden_shape[2]
+
+  output_weights = tf.compat.v1.get_variable(
+      "cls/squad/output_weights", [2, hidden_size],
+      initializer=tf.compat.v1.truncated_normal_initializer(stddev=0.02))
+
+  output_bias = tf.compat.v1.get_variable(
+      "cls/squad/output_bias", [2], initializer=tf.compat.v1.zeros_initializer())
+
+  final_hidden_matrix = tf.reshape(final_hidden,
+                                   [batch_size * seq_length, hidden_size])
+  logits = tf.matmul(final_hidden_matrix, output_weights, transpose_b=True)
+  logits = tf.nn.bias_add(logits, output_bias)
+
+  logits = tf.reshape(logits, [batch_size, seq_length, 2])
+  logits = tf.transpose(a=logits, perm=[2, 0, 1])
+
+  unstacked_logits = tf.unstack(logits, axis=0)
+
+  (start_logits, end_logits) = (unstacked_logits[0], unstacked_logits[1])
+  start = tf.identity(start_logits, name = "start_logits")
+  end = tf.identity(end_logits, name = "end_logits")
+
+  return (start, end)
+
+
+def create_model_top(bert_config, is_training, input_ids, input_mask, segment_ids,
+                     use_one_hot_embeddings, frozen_graph=None):
+    if frozen_graph and not is_training:
+        print("reading***** From *** pb", flush=True)
+        inputs ={'input_ids': input_ids,
+                 'input_mask': input_mask,
+                 'segment_ids': segment_ids}
+
+        with tf.io.gfile.GFile(frozen_graph, "rb") as f:
+            graph_def = tf.compat.v1.GraphDef()
+            graph_def.ParseFromString(f.read())
+
+        all_node_name = [n.name for n in graph_def.node]
+
+        outputs = []
+
+        candidate_outputs_1 = ['unstacked_logits:0', 'unstacked_logits:1']
+        candidate_outputs_2 = ['start_logits:0', 'end_logits:0']
+
+        if "unstacked_logits" in all_node_name:
+          outputs = candidate_outputs_1
+
+        if "start_logits" in all_node_name:
+          outputs = candidate_outputs_2
+
+        return tf.graph_util.import_graph_def(graph_def, inputs, outputs, name="")
+
+    return create_model(
+        bert_config=bert_config,
+        is_training=is_training,
+        input_ids=input_ids,
+        input_mask=input_mask,
+        segment_ids=segment_ids,
+        use_one_hot_embeddings=use_one_hot_embeddings)
+
+
+def model_fn_builder(bert_config, init_checkpoint, learning_rate,
+                     num_train_steps, num_warmup_steps, use_tpu,
+                     use_one_hot_embeddings, frozen_graph):
+  """Returns `model_fn` closure for TPUEstimator."""
+
+  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
+    """The `model_fn` for TPUEstimator."""
+
+    tf.compat.v1.logging.info("*** Features ***")
+    for name in sorted(features.keys()):
+      tf.compat.v1.logging.info("  name = %s, shape = %s" % (name, features[name].shape))
+
+    unique_ids = features["unique_ids"]
+    input_ids = features["input_ids"]
+    input_mask = features["input_mask"]
+    segment_ids = features["segment_ids"]
+
+    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+    if bert_config.precision == "bfloat16" :
+      with tf.compat.v1.tpu.bfloat16_scope():
+        (start_logits, end_logits) = create_model_top(
+            bert_config=bert_config,
+            is_training=is_training,
+            input_ids=input_ids,
+            input_mask=input_mask,
+            segment_ids=segment_ids,
+            use_one_hot_embeddings=use_one_hot_embeddings,
+            frozen_graph=frozen_graph)
+      start_logits = tf.cast(start_logits, tf.float32)
+      end_logits = tf.cast(end_logits, tf.float32)
+    else :
+        (start_logits, end_logits) = create_model_top(
+            bert_config=bert_config,
+            is_training=is_training,
+            input_ids=input_ids,
+            input_mask=input_mask,
+            segment_ids=segment_ids,
+            use_one_hot_embeddings=use_one_hot_embeddings,
+            frozen_graph=frozen_graph)
+
+    tvars = tf.compat.v1.trainable_variables()
+
+    initialized_variable_names = {}
+    scaffold_fn = None
+    if init_checkpoint:
+      (assignment_map, initialized_variable_names
+      ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint, "SQuAD")
+      if use_tpu:
+
+        def tpu_scaffold():
+          tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+          return tf.compat.v1.train.Scaffold()
+
+        scaffold_fn = tpu_scaffold
+      else:
+        tf.compat.v1.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+    tf.compat.v1.logging.info("**** Trainable Variables ****")
+    for var in tvars:
+      init_string = ""
+      if var.name in initialized_variable_names:
+        init_string = ", *INIT_FROM_CKPT*"
+      tf.compat.v1.logging.info("  name = %s, shape = %s%s", var.name, var.shape,
+                      init_string)
+
+    output_spec = None
+    if mode == tf.estimator.ModeKeys.TRAIN:
+      seq_length = modeling.get_shape_list(input_ids)[1]
+
+      def compute_loss(logits, positions):
+        one_hot_positions = tf.one_hot(
+            positions, depth=seq_length, dtype=tf.float32)
+        log_probs = tf.nn.log_softmax(logits, axis=-1)
+        loss = -tf.reduce_mean(
+            input_tensor=tf.reduce_sum(input_tensor=one_hot_positions * log_probs, axis=-1))
+        return loss
+
+      start_positions = features["start_positions"]
+      end_positions = features["end_positions"]
+
+      start_loss = compute_loss(start_logits, start_positions)
+      end_loss = compute_loss(end_logits, end_positions)
+
+      total_loss = (start_loss + end_loss) / 2.0
+
+      train_op = optimization.create_optimizer(
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu)
+
+      log_hook = bf.logTheLossHook(total_loss, n=3)
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode,
+          loss=total_loss,
+          train_op=train_op,
+          training_hooks=[log_hook],
+          scaffold_fn=scaffold_fn)
+    elif mode == tf.estimator.ModeKeys.PREDICT:
+      predictions = {
+          "unique_ids": unique_ids,
+          "start_logits": start_logits,
+          "end_logits": end_logits,
+      }
+      output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
+          mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)
+    else:
+      raise ValueError(
+          "Only TRAIN and PREDICT modes are supported: %s" % (mode))
+
+    return output_spec
+
+  return model_fn
+
+
+def input_fn_builder(input_file, seq_length, is_training, drop_remainder):
+  """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+  name_to_features = {
+      "unique_ids": tf.io.FixedLenFeature([], tf.int64),
+      "input_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "input_mask": tf.io.FixedLenFeature([seq_length], tf.int64),
+      "segment_ids": tf.io.FixedLenFeature([seq_length], tf.int64),
+  }
+
+  if is_training:
+    name_to_features["start_positions"] = tf.io.FixedLenFeature([], tf.int64)
+    name_to_features["end_positions"] = tf.io.FixedLenFeature([], tf.int64)
+
+  def _decode_record(record, name_to_features):
+    """Decodes a record to a TensorFlow example."""
+    example = tf.io.parse_single_example(serialized=record, features=name_to_features)
+
+    # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+    # So cast all int64 to int32.
+    for name in list(example.keys()):
+      t = example[name]
+      if t.dtype == tf.int64:
+        t = tf.cast(t, dtype=tf.int32)
+      example[name] = t
+
+    return example
+
+  def input_fn(params):
+    """The actual input function."""
+    batch_size = params["batch_size"]
+
+    # For training, we want a lot of parallel reading and shuffling.
+    # For eval, we want no shuffling and parallel reading doesn't matter.
+    d = tf.data.TFRecordDataset(input_file)
+    if is_training:
+      d = d.repeat()
+      d = d.shuffle(buffer_size=100)
+
+    d = d.apply(
+        tf.data.experimental.map_and_batch(
+            lambda record: _decode_record(record, name_to_features),
+            batch_size=batch_size,
+            drop_remainder=drop_remainder))
+
+    return d
+
+  return input_fn
+
+
+RawResult = collections.namedtuple("RawResult",
+                                   ["unique_id", "start_logits", "end_logits"])
+
+
+def write_predictions(all_examples, all_features, all_results, n_best_size,
+                      max_answer_length, do_lower_case, output_prediction_file,
+                      output_nbest_file, output_null_log_odds_file):
+  """Write final predictions to the json file and log-odds of null if needed."""
+  tf.compat.v1.logging.info("Writing predictions to: %s" % (output_prediction_file))
+  tf.compat.v1.logging.info("Writing nbest to: %s" % (output_nbest_file))
+
+  example_index_to_features = collections.defaultdict(list)
+  for feature in all_features:
+    example_index_to_features[feature.example_index].append(feature)
+
+  unique_id_to_result = {}
+  for result in all_results:
+    unique_id_to_result[result.unique_id] = result
+
+  _PrelimPrediction = collections.namedtuple(  # pylint: disable=invalid-name
+      "PrelimPrediction",
+      ["feature_index", "start_index", "end_index", "start_logit", "end_logit"])
+
+  all_predictions = collections.OrderedDict()
+  all_nbest_json = collections.OrderedDict()
+  scores_diff_json = collections.OrderedDict()
+
+  for (example_index, example) in enumerate(all_examples):
+    features = example_index_to_features[example_index]
+
+    prelim_predictions = []
+    # keep track of the minimum score of null start+end of position 0
+    score_null = 1000000  # large and positive
+    min_null_feature_index = 0  # the paragraph slice with min mull score
+    null_start_logit = 0  # the start logit at the slice with min null score
+    null_end_logit = 0  # the end logit at the slice with min null score
+    for (feature_index, feature) in enumerate(features):
+      result = unique_id_to_result[feature.unique_id]
+      start_indexes = _get_best_indexes(result.start_logits, n_best_size)
+      end_indexes = _get_best_indexes(result.end_logits, n_best_size)
+      # if we could have irrelevant answers, get the min score of irrelevant
+      if FLAGS.version_2_with_negative:
+        feature_null_score = result.start_logits[0] + result.end_logits[0]
+        if feature_null_score < score_null:
+          score_null = feature_null_score
+          min_null_feature_index = feature_index
+          null_start_logit = result.start_logits[0]
+          null_end_logit = result.end_logits[0]
+      for start_index in start_indexes:
+        for end_index in end_indexes:
+          # We could hypothetically create invalid predictions, e.g., predict
+          # that the start of the span is in the question. We throw out all
+          # invalid predictions.
+          if start_index >= len(feature.tokens):
+            continue
+          if end_index >= len(feature.tokens):
+            continue
+          if start_index not in feature.token_to_orig_map:
+            continue
+          if end_index not in feature.token_to_orig_map:
+            continue
+          if not feature.token_is_max_context.get(start_index, False):
+            continue
+          if end_index < start_index:
+            continue
+          length = end_index - start_index + 1
+          if length > max_answer_length:
+            continue
+          prelim_predictions.append(
+              _PrelimPrediction(
+                  feature_index=feature_index,
+                  start_index=start_index,
+                  end_index=end_index,
+                  start_logit=result.start_logits[start_index],
+                  end_logit=result.end_logits[end_index]))
+
+    if FLAGS.version_2_with_negative:
+      prelim_predictions.append(
+          _PrelimPrediction(
+              feature_index=min_null_feature_index,
+              start_index=0,
+              end_index=0,
+              start_logit=null_start_logit,
+              end_logit=null_end_logit))
+    prelim_predictions = sorted(
+        prelim_predictions,
+        key=lambda x: (x.start_logit + x.end_logit),
+        reverse=True)
+
+    _NbestPrediction = collections.namedtuple(  # pylint: disable=invalid-name
+        "NbestPrediction", ["text", "start_logit", "end_logit"])
+
+    seen_predictions = {}
+    nbest = []
+    for pred in prelim_predictions:
+      if len(nbest) >= n_best_size:
+        break
+      feature = features[pred.feature_index]
+      if pred.start_index > 0:  # this is a non-null prediction
+        tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
+        orig_doc_start = feature.token_to_orig_map[pred.start_index]
+        orig_doc_end = feature.token_to_orig_map[pred.end_index]
+        orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
+        tok_text = " ".join(tok_tokens)
+
+        # De-tokenize WordPieces that have been split off.
+        tok_text = tok_text.replace(" ##", "")
+        tok_text = tok_text.replace("##", "")
+
+        # Clean whitespace
+        tok_text = tok_text.strip()
+        tok_text = " ".join(tok_text.split())
+        orig_text = " ".join(orig_tokens)
+
+        final_text = get_final_text(tok_text, orig_text, do_lower_case)
+        if final_text in seen_predictions:
+          continue
+
+        seen_predictions[final_text] = True
+      else:
+        final_text = ""
+        seen_predictions[final_text] = True
+
+      nbest.append(
+          _NbestPrediction(
+              text=final_text,
+              start_logit=pred.start_logit,
+              end_logit=pred.end_logit))
+
+    # if we didn't inlude the empty option in the n-best, inlcude it
+    if FLAGS.version_2_with_negative:
+      if "" not in seen_predictions:
+        nbest.append(
+            _NbestPrediction(
+                text="", start_logit=null_start_logit,
+                end_logit=null_end_logit))
+    # In very rare edge cases we could have no valid predictions. So we
+    # just create a nonce prediction in this case to avoid failure.
+    if not nbest:
+      nbest.append(
+          _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
+
+    assert len(nbest) >= 1
+
+    total_scores = []
+    best_non_null_entry = None
+    for entry in nbest:
+      total_scores.append(entry.start_logit + entry.end_logit)
+      if not best_non_null_entry:
+        if entry.text:
+          best_non_null_entry = entry
+
+    probs = _compute_softmax(total_scores)
+
+    nbest_json = []
+    for (i, entry) in enumerate(nbest):
+      output = collections.OrderedDict()
+      output["text"] = entry.text
+      output["probability"] = probs[i]
+      output["start_logit"] = entry.start_logit
+      output["end_logit"] = entry.end_logit
+      nbest_json.append(output)
+
+    assert len(nbest_json) >= 1
+
+    if not FLAGS.version_2_with_negative:
+      all_predictions[example.qas_id] = nbest_json[0]["text"]
+    else:
+      # predict "" iff the null score - the score of best non-null > threshold
+      score_diff = score_null - best_non_null_entry.start_logit - (
+          best_non_null_entry.end_logit)
+      scores_diff_json[example.qas_id] = score_diff
+      if score_diff > FLAGS.null_score_diff_threshold:
+        all_predictions[example.qas_id] = ""
+      else:
+        all_predictions[example.qas_id] = best_non_null_entry.text
+
+    all_nbest_json[example.qas_id] = nbest_json
+
+  with tf.io.gfile.GFile(output_prediction_file, "w") as writer:
+    writer.write(json.dumps(all_predictions, indent=4) + "\n")
+
+  with tf.io.gfile.GFile(output_nbest_file, "w") as writer:
+    writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
+
+  if FLAGS.version_2_with_negative:
+    with tf.io.gfile.GFile(output_null_log_odds_file, "w") as writer:
+      writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
+
+
+def get_final_text(pred_text, orig_text, do_lower_case):
+  """Project the tokenized prediction back to the original text."""
+
+  # When we created the data, we kept track of the alignment between original
+  # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
+  # now `orig_text` contains the span of our original text corresponding to the
+  # span that we predicted.
+  #
+  # However, `orig_text` may contain extra characters that we don't want in
+  # our prediction.
+  #
+  # For example, let's say:
+  #   pred_text = steve smith
+  #   orig_text = Steve Smith's
+  #
+  # We don't want to return `orig_text` because it contains the extra "'s".
+  #
+  # We don't want to return `pred_text` because it's already been normalized
+  # (the SQuAD eval script also does punctuation stripping/lower casing but
+  # our tokenizer does additional normalization like stripping accent
+  # characters).
+  #
+  # What we really want to return is "Steve Smith".
+  #
+  # Therefore, we have to apply a semi-complicated alignment heruistic between
+  # `pred_text` and `orig_text` to get a character-to-charcter alignment. This
+  # can fail in certain cases in which case we just return `orig_text`.
+
+  def _strip_spaces(text):
+    ns_chars = []
+    ns_to_s_map = collections.OrderedDict()
+    for (i, c) in enumerate(text):
+      if c == " ":
+        continue
+      ns_to_s_map[len(ns_chars)] = i
+      ns_chars.append(c)
+    ns_text = "".join(ns_chars)
+    return (ns_text, ns_to_s_map)
+
+  # We first tokenize `orig_text`, strip whitespace from the result
+  # and `pred_text`, and check if they are the same length. If they are
+  # NOT the same length, the heuristic has failed. If they are the same
+  # length, we assume the characters are one-to-one aligned.
+  tokenizer = tokenization.BasicTokenizer(do_lower_case=do_lower_case)
+
+  tok_text = " ".join(tokenizer.tokenize(orig_text))
+
+  start_position = tok_text.find(pred_text)
+  if start_position == -1:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info(
+          "Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
+    return orig_text
+  end_position = start_position + len(pred_text) - 1
+
+  (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
+  (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
+
+  if len(orig_ns_text) != len(tok_ns_text):
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Length not equal after stripping spaces: '%s' vs '%s'",
+                      orig_ns_text, tok_ns_text)
+    return orig_text
+
+  # We then project the characters in `pred_text` back to `orig_text` using
+  # the character-to-character alignment.
+  tok_s_to_ns_map = {}
+  for (i, tok_index) in six.iteritems(tok_ns_to_s_map):
+    tok_s_to_ns_map[tok_index] = i
+
+  orig_start_position = None
+  if start_position in tok_s_to_ns_map:
+    ns_start_position = tok_s_to_ns_map[start_position]
+    if ns_start_position in orig_ns_to_s_map:
+      orig_start_position = orig_ns_to_s_map[ns_start_position]
+
+  if orig_start_position is None:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Couldn't map start position")
+    return orig_text
+
+  orig_end_position = None
+  if end_position in tok_s_to_ns_map:
+    ns_end_position = tok_s_to_ns_map[end_position]
+    if ns_end_position in orig_ns_to_s_map:
+      orig_end_position = orig_ns_to_s_map[ns_end_position]
+
+  if orig_end_position is None:
+    if FLAGS.verbose_logging:
+      tf.compat.v1.logging.info("Couldn't map end position")
+    return orig_text
+
+  output_text = orig_text[orig_start_position:(orig_end_position + 1)]
+  return output_text
+
+
+def _get_best_indexes(logits, n_best_size):
+  """Get the n-best logits from a list."""
+  index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)
+
+  best_indexes = []
+  for i in range(len(index_and_score)):
+    if i >= n_best_size:
+      break
+    best_indexes.append(index_and_score[i][0])
+  return best_indexes
+
+
+def _compute_softmax(scores):
+  """Compute softmax probability over raw logits."""
+  if not scores:
+    return []
+
+  max_score = None
+  for score in scores:
+    if max_score is None or score > max_score:
+      max_score = score
+
+  exp_scores = []
+  total_sum = 0.0
+  for score in scores:
+    x = math.exp(score - max_score)
+    exp_scores.append(x)
+    total_sum += x
+
+  probs = []
+  for score in exp_scores:
+    probs.append(score / total_sum)
+  return probs
+
+
+class FeatureWriter(object):
+  """Writes InputFeature to TF example file."""
+
+  def __init__(self, filename, is_training):
+    self.filename = filename
+    self.is_training = is_training
+    self.num_features = 0
+    self._writer = tf.io.TFRecordWriter(filename)
+
+  def process_feature(self, feature):
+    """Write a InputFeature to the TFRecordWriter as a tf.train.Example."""
+    self.num_features += 1
+
+    def create_int_feature(values):
+      feature = tf.train.Feature(
+          int64_list=tf.train.Int64List(value=list(values)))
+      return feature
+
+    features = collections.OrderedDict()
+    features["unique_ids"] = create_int_feature([feature.unique_id])
+    features["input_ids"] = create_int_feature(feature.input_ids)
+    features["input_mask"] = create_int_feature(feature.input_mask)
+    features["segment_ids"] = create_int_feature(feature.segment_ids)
+
+    if self.is_training:
+      features["start_positions"] = create_int_feature([feature.start_position])
+      features["end_positions"] = create_int_feature([feature.end_position])
+      impossible = 0
+      if feature.is_impossible:
+        impossible = 1
+      features["is_impossible"] = create_int_feature([impossible])
+
+    tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+    self._writer.write(tf_example.SerializeToString())
+
+  def close(self):
+    self._writer.close()
+
+
+def validate_flags_or_throw(bert_config):
+  """Validate the input FLAGS or throw an exception."""
+  tokenization.validate_case_matches_checkpoint(FLAGS.do_lower_case,
+                                                FLAGS.init_checkpoint)
+
+  if not FLAGS.do_train and not FLAGS.do_predict:
+    raise ValueError("At least one of `do_train` or `do_predict` must be True.")
+
+  if FLAGS.do_train:
+    if not FLAGS.train_file:
+      raise ValueError(
+          "If `do_train` is True, then `train_file` must be specified.")
+  if FLAGS.do_predict:
+    if not FLAGS.predict_file:
+      raise ValueError(
+          "If `do_predict` is True, then `predict_file` must be specified.")
+
+  if FLAGS.precision:
+    bert_config.precision = FLAGS.precision
+
+  if FLAGS.max_seq_length > bert_config.max_position_embeddings:
+    raise ValueError(
+        "Cannot use sequence length %d because the BERT model "
+        "was only trained up to sequence length %d" %
+        (FLAGS.max_seq_length, bert_config.max_position_embeddings))
+
+  if FLAGS.max_seq_length <= FLAGS.max_query_length + 3:
+    raise ValueError(
+        "The max_seq_length (%d) must be greater than max_query_length "
+        "(%d) + 3" % (FLAGS.max_seq_length, FLAGS.max_query_length))
+
+def do_benchmark():
+  import numpy as np
+  import threading
+  if FLAGS.weight_sharing:
+    if not FLAGS.num_cores_per_socket:
+      raise ValueError(
+          "Weight sharing requires number of cores per socket info.")
+    if not os.getenv("OMP_NUM_THREADS"):
+      raise ValueError(
+          "Weight sharing requires OMP_NUM_THREADS environment variable setting.")
+    num_instances = int(
+        FLAGS.num_cores_per_socket //
+        int(os.getenv("OMP_NUM_THREADS")))
+  else:
+    num_instances = 1
+  if num_instances < 1:
+    num_instances = 1
+  throughput_list = [None] * num_instances
+  # TODO(intel-tf): Generalize the values of seq, query, and paragraph lengths.
+  # Hard setting for running benchmark without data layer now.
+  MAX_SEQ_LENGTH = 384
+  MIN_QUERY_LENGTH = 10
+  MAX_QUERY_LENGTH = 20
+  MIN_PARAGRAPH_LENGTH = 120
+  MAX_PARAGRAPH_LENGTH = 300
+  VOCAB_SIZE = 30522
+
+  def create_feed_dict():
+    question_length = np.random.randint(MIN_QUERY_LENGTH, MAX_QUERY_LENGTH)
+    paragraph_length = np.random.randint(
+        MIN_PARAGRAPH_LENGTH, MAX_PARAGRAPH_LENGTH)
+    input_ids = np.random.randint(
+        1, VOCAB_SIZE, [
+            question_length + paragraph_length])
+    input_ids = np.pad(input_ids, (0, MAX_SEQ_LENGTH - len(input_ids)))
+    input_mask = np.ones(question_length + paragraph_length)
+    input_mask = np.pad(input_mask, (0, MAX_SEQ_LENGTH - len(input_mask)))
+    segment_ids = np.ones(paragraph_length)
+    segment_ids = np.pad(segment_ids, (question_length,
+                                       MAX_SEQ_LENGTH - (question_length + paragraph_length)))
+    input_ids = input_ids[None, :]
+    input_mask = input_mask[None, :]
+    segment_ids = segment_ids[None, :]
+    for _ in range(FLAGS.predict_batch_size - 1):
+      question_length = np.random.randint(MIN_QUERY_LENGTH, MAX_QUERY_LENGTH)
+      paragraph_length = np.random.randint(
+          MIN_PARAGRAPH_LENGTH, MAX_PARAGRAPH_LENGTH)
+      _input_ids = np.random.randint(
+          1, VOCAB_SIZE, [
+              question_length + paragraph_length])
+      _input_ids = np.pad(_input_ids, (0, MAX_SEQ_LENGTH - len(_input_ids)))
+      _input_mask = np.ones(question_length + paragraph_length)
+      _input_mask = np.pad(_input_mask, (0, MAX_SEQ_LENGTH - len(_input_mask)))
+      _segment_ids = np.ones(paragraph_length)
+      _segment_ids = np.pad(_segment_ids, (question_length,
+                                           MAX_SEQ_LENGTH - (question_length + paragraph_length)))
+      input_ids = np.vstack([input_ids, _input_ids])
+      input_mask = np.vstack([input_mask, _input_mask])
+      segment_ids = np.vstack([segment_ids, _segment_ids])
+    return {
+        'input_ids:0': input_ids,
+        'input_mask:0': input_mask,
+        'segment_ids:0': segment_ids}
+
+  def run_model(sess, tid):
+    all_tensor_name = [tensor.name for op in sess.graph.get_operations() for tensor in op.values()]
+    
+    outputs = []
+
+    candidate_outputs_1 = ['unstacked_logits:0', 'unstacked_logits:1']
+    candidate_outputs_2 = ['start_logits:0', 'end_logits:0']
+
+    if candidate_outputs_1[0] in all_tensor_name and candidate_outputs_1[1] in all_tensor_name:
+      outputs = candidate_outputs_1
+
+    if candidate_outputs_2[0] in all_tensor_name and candidate_outputs_2[1] in all_tensor_name:
+      outputs = candidate_outputs_2
+
+    feed_dict = create_feed_dict()
+    for step in range(FLAGS.warmup_steps):
+      _ = sess.run(outputs, feed_dict=feed_dict)
+    t1 = time.time()
+    for step in range(FLAGS.steps):
+      _ = sess.run(outputs, feed_dict=feed_dict)
+    t2 = time.time()
+    throughput_list[tid] = FLAGS.predict_batch_size * FLAGS.steps / (t2 - t1)
+    print('Thread {0} has throughput {1}'.format(tid + 1, throughput_list[tid]))
+
+  # Override inter_op_parallelism_threads when weight sharing on
+  if FLAGS.weight_sharing:
+    FLAGS.inter_op_parallelism_threads = -1
+  config = tf.compat.v1.ConfigProto(
+      intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads,
+      inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads)
+  if FLAGS.precision == "fp16":
+    print("Using grappler auto-mixed precision")
+    config.graph_options.rewrite_options.auto_mixed_precision = (
+              rewriter_config_pb2.RewriterConfig.ON)
+  config.graph_options.rewrite_options.constant_folding = (
+          rewriter_config_pb2.RewriterConfig.OFF)
+
+  graph_def = tf.compat.v1.GraphDef()
+  with open(FLAGS.input_graph, "rb") as f:
+    graph_def.ParseFromString(f.read())
+  graph = tf.Graph()
+  with graph.as_default():
+    tf.import_graph_def(graph_def, name="")
+  sess = tf.compat.v1.Session(graph=graph, config=config)
+  if num_instances == 1:
+    run_model(sess, 0)
+  else:
+    threads = [None] * num_instances
+    for tid in range(num_instances):
+      thread = threading.Thread(target=run_model, args=(sess, tid))
+      threads[tid] = thread
+      thread.start()
+    for thread in threads:
+      thread.join()
+
+  print('Total throughput (examples/sec): {0}'.format(sum(throughput_list)))
+
+def main(_):
+  tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
+
+  print("Running with " + FLAGS.precision + " precision")
+
+  if FLAGS.mode == 'benchmark' and FLAGS.input_graph:
+    tf.compat.v1.disable_eager_execution()
+    do_benchmark()
+    return
+
+  if (FLAGS.accum_steps >1 ):
+    tf.compat.v1.logging.info(" Accum steps not yet supported in SQuAD")
+    exit(0)
+
+  if FLAGS.profile:
+    tf.compat.v1.disable_eager_execution()
+
+  bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+  bert_config.experimental_gelu = FLAGS.experimental_gelu
+  bert_config.optimized_softmax = FLAGS.optimized_softmax
+
+  validate_flags_or_throw(bert_config)
+
+  tf.io.gfile.makedirs(FLAGS.output_dir)
+
+  tokenizer = tokenization.FullTokenizer(
+      vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+  tpu_cluster_resolver = None
+  if FLAGS.use_tpu and FLAGS.tpu_name:
+    tpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
+        FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+  session_config = tf.compat.v1.ConfigProto(
+      inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
+      intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads,
+      allow_soft_placement=True)
+  if bert_config.precision == "fp16":
+      session_config.graph_options.rewrite_options.auto_mixed_precision = (
+              rewriter_config_pb2.RewriterConfig.ON)
+  session_config.graph_options.rewrite_options.constant_folding = (
+          rewriter_config_pb2.RewriterConfig.OFF)
+
+  is_per_host = tf.compat.v1.estimator.tpu.InputPipelineConfig.PER_HOST_V2
+  run_config = tf.compat.v1.estimator.tpu.RunConfig(
+      cluster=tpu_cluster_resolver,
+      master=FLAGS.master,
+      model_dir=FLAGS.output_dir,
+      save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+      session_config=session_config,
+      tpu_config=tf.compat.v1.estimator.tpu.TPUConfig(
+          iterations_per_loop=FLAGS.iterations_per_loop,
+          num_shards=FLAGS.num_tpu_cores,
+          per_host_input_for_training=is_per_host))
+
+  train_examples = None
+  num_train_steps = None
+  num_warmup_steps = None
+  if FLAGS.do_train:
+    train_examples = read_squad_examples(
+        input_file=FLAGS.train_file, is_training=True)
+    num_train_steps = int(
+        len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+    num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+    # Pre-shuffle the input to avoid having to make a very large shuffle
+    # buffer in in the `input_fn`.
+    rng = random.Random(12345)
+    rng.shuffle(train_examples)
+
+  model_fn = model_fn_builder(
+      bert_config=bert_config,
+      init_checkpoint=FLAGS.init_checkpoint,
+      learning_rate=FLAGS.learning_rate,
+      num_train_steps=num_train_steps,
+      num_warmup_steps=num_warmup_steps,
+      use_tpu=FLAGS.use_tpu,
+      use_one_hot_embeddings=FLAGS.use_tpu,
+      frozen_graph=FLAGS.input_graph)
+
+  # If TPU is not available, this will fall back to normal Estimator on CPU
+  # or GPU.
+  estimator = tf.compat.v1.estimator.tpu.TPUEstimator(
+      use_tpu=FLAGS.use_tpu,
+      model_fn=model_fn,
+      config=run_config,
+      train_batch_size=FLAGS.train_batch_size,
+      predict_batch_size=FLAGS.predict_batch_size)
+
+  if FLAGS.do_train:
+    # We write to a temporary file to avoid storing very large constant tensors
+    # in memory.
+    train_writer = FeatureWriter(
+        filename=os.path.join(FLAGS.output_dir, "train.tf_record"),
+        is_training=True)
+    convert_examples_to_features(
+        examples=train_examples,
+        tokenizer=tokenizer,
+        max_seq_length=FLAGS.max_seq_length,
+        doc_stride=FLAGS.doc_stride,
+        max_query_length=FLAGS.max_query_length,
+        is_training=True,
+        output_fn=train_writer.process_feature)
+    train_writer.close()
+
+    tf.compat.v1.logging.info("***** Running training *****")
+    tf.compat.v1.logging.info("  Num orig examples = %d", len(train_examples))
+    tf.compat.v1.logging.info("  Num split examples = %d", train_writer.num_features)
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.train_batch_size)
+    tf.compat.v1.logging.info("  Num steps = %d", num_train_steps)
+    del train_examples
+
+    train_input_fn = input_fn_builder(
+        input_file=train_writer.filename,
+        seq_length=FLAGS.max_seq_length,
+        is_training=True,
+        drop_remainder=True)
+
+    # Added to profile the run
+    if FLAGS.profile ==True :
+      tf.compat.v1.logging.info("***** Running training with profiler*****")
+      hooks = [tf.compat.v1.train.ProfilerHook(save_steps=3, output_dir=FLAGS.output_dir,
+                                               show_memory=True)]
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps, hooks=hooks)
+    else:
+      estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+  if FLAGS.do_predict:
+    eval_examples = read_squad_examples(
+        input_file=FLAGS.predict_file, is_training=False)
+
+    eval_writer = FeatureWriter(
+        filename=os.path.join(FLAGS.output_dir, "eval.tf_record"),
+        is_training=False)
+    eval_features = []
+
+    def append_feature(feature):
+      eval_features.append(feature)
+      eval_writer.process_feature(feature)
+
+    convert_examples_to_features(
+        examples=eval_examples,
+        tokenizer=tokenizer,
+        max_seq_length=FLAGS.max_seq_length,
+        doc_stride=FLAGS.doc_stride,
+        max_query_length=FLAGS.max_query_length,
+        is_training=False,
+        output_fn=append_feature)
+    eval_writer.close()
+
+    tf.compat.v1.logging.info("***** Running predictions *****")
+    tf.compat.v1.logging.info("  Num orig examples = %d", len(eval_examples))
+    tf.compat.v1.logging.info("  Num split examples = %d", len(eval_features))
+    tf.compat.v1.logging.info("  Batch size = %d", FLAGS.predict_batch_size)
+
+    all_results = []
+
+    predict_input_fn = input_fn_builder(
+        input_file=eval_writer.filename,
+        seq_length=FLAGS.max_seq_length,
+        is_training=False,
+        drop_remainder=False)
+
+    # If running eval on the TPU, you will need to specify the number of
+    # steps.
+    tf.compat.v1.disable_eager_execution()
+    warmup_steps = FLAGS.warmup_steps
+    total_steps = FLAGS.warmup_steps + FLAGS.steps
+    if FLAGS.mode == 'benchmark':
+      hooks = [UpdateGlobalStepHook(),
+               StopAtStepHook(total_steps)]
+    elif FLAGS.mode == 'profile':
+      hooks = [UpdateGlobalStepHook(),
+               StopAtStepHook(total_steps),
+               ProfilerHook(save_steps=10, output_dir=FLAGS.output_dir)]
+    else:
+      hooks = []
+    num_warmup_examples = warmup_steps * FLAGS.predict_batch_size
+    num_processed_examples = 0
+    all_results = []
+    for result in estimator.predict(
+        predict_input_fn, yield_single_examples=True,
+        hooks=hooks):
+      num_processed_examples += 1
+      if FLAGS.mode in ('benchmark', 'profile'):
+        if num_processed_examples % 10 == 0:
+          tf.compat.v1.logging.info("Processed #examples: %d" % (num_processed_examples))
+        if num_processed_examples == (num_warmup_examples):
+          start = time.time()
+      elif FLAGS.mode == 'accuracy':
+        if len(all_results) % 10 == 0:
+          tf.compat.v1.logging.info("Processing example: %d" % (len(all_results)))
+        unique_id = int(result["unique_ids"])
+        start_logits = [float(x) for x in result["start_logits"].flat]
+        end_logits = [float(x) for x in result["end_logits"].flat]
+        all_results.append(
+            RawResult(
+                unique_id=unique_id,
+                start_logits=start_logits,
+                end_logits=end_logits))
+    if FLAGS.mode in ('benchmark', 'profile'):
+      end = time.time()
+      num_benchmark_examples = num_processed_examples-num_warmup_examples
+      benchmark_duration = end - start
+      print("Total examples: %d, Warmup examples: %d, "
+            "Benchmark examples: %d, Benchmark time: %f secs"
+            % (num_processed_examples, num_warmup_examples,
+            num_benchmark_examples, benchmark_duration))
+      print("Throughput: %3.2f"
+            % (float(num_benchmark_examples)/float(benchmark_duration)))
+    if FLAGS.mode == 'accuracy':
+      output_prediction_file = os.path.join(FLAGS.output_dir, "predictions.json")
+      output_nbest_file = os.path.join(FLAGS.output_dir, "nbest_predictions.json")
+      output_null_log_odds_file = os.path.join(FLAGS.output_dir, "null_odds.json")
+
+      write_predictions(eval_examples, eval_features, all_results,
+                        FLAGS.n_best_size, FLAGS.max_answer_length,
+                        FLAGS.do_lower_case, output_prediction_file,
+                        output_nbest_file, output_null_log_odds_file)
+
+if __name__ == "__main__":
+  flags.mark_flag_as_required("vocab_file")
+  flags.mark_flag_as_required("bert_config_file")
+  flags.mark_flag_as_required("output_dir")
+  tf.compat.v1.app.run()
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/generic_ops.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/generic_ops.py
index 70f9a0ea9..31825ebe6 100755
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/generic_ops.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/generic_ops.py
@@ -82,6 +82,14 @@ def softmax(scores, axis=None):
 
 def layer_norm(inputs, begin_norm_axis, begin_params_axis, scope):
     lnorm = tf.keras.layers.LayerNormalization(dtype=get_keras_policy())
+
+    # Try to use ITEX first
+    try:
+      import intel_extension_for_tensorflow as itex
+      lnorm = itex.ops.LayerNormalization(dtype=_rprecision)
+    except ImportError:
+      pass
+
     return lnorm(inputs)
 
 "Moved from modeling.py"
@@ -96,8 +104,17 @@ def gelu(x):
   Returns:
     `x` with the GELU activation applied.
   """
-  if _use_experimental_gelu:
-    return tf.nn.gelu(features=x, approximate=True)
+  if _use_experimental_gelu :
+    gelu_func = tf.nn.gelu
+
+    # Try to use ITXE first.
+    try:
+      import intel_extension_for_tensorflow as itex
+      gelu_func = itex.ops.gelu
+    except ImportError:
+      pass
+
+    return gelu_func(features=x, approximate=True)
   else:
     x = i_cast(x)
     cdf = 0.5 * (1.0 + tf.tanh(
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/optimization.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/optimization.py
index d0b88b86c..d9dad21f6 100644
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/optimization.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/optimization.py
@@ -22,7 +22,7 @@
 import tensorflow as tf
 
 
-def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=0):
+def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=0, use_gpu=False):
   """Creates an optimizer training op."""
   global_step = tf.compat.v1.train.get_or_create_global_step()
 
@@ -58,17 +58,21 @@ def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_ste
   # is how the model was trained (note that the Adam m/v variables are NOT
   # loaded from init_checkpoint.)
   use_itex_optimizer = False
-  try:
-    import intel_extension_for_tensorflow as itex
-    optimizer = itex.ops.AdamWithWeightDecayOptimizer(
-        learning_rate=learning_rate,
-        weight_decay_rate=0.01,
-        beta_1=0.9,
-        beta_2=0.999,
-        epsilon=1e-6,
-        exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
-    use_itex_optimizer = True
-  except:
+  if use_gpu:
+    try:
+      import intel_extension_for_tensorflow as itex
+      optimizer = itex.ops.AdamWithWeightDecayOptimizer(
+          learning_rate=learning_rate,
+          weight_decay_rate=0.01,
+          beta_1=0.9,
+          beta_2=0.999,
+          epsilon=1e-6,
+          exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
+      use_itex_optimizer = True
+    except:
+      print("Load plugin Intel(R) Extension for Tensorflow* failed.\n")
+
+  if not use_itex_optimizer:
     optimizer = AdamWeightDecayOptimizer(
         learning_rate=learning_rate,
         weight_decay_rate=0.01,
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier.py
index 95c2f18c9..efb5a3f6b 100644
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier.py
@@ -183,6 +183,8 @@ def __init__(self, guid, text_a, text_b=None, label=None):
     "[Optional] If true, use more experimental op(gelu) in model."
     "           Be careful this flag will crash model with incompatible TF.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 class PaddingInputExample(object):
   """Fake example so the num input examples is a multiple of the batch size.
 
@@ -730,7 +732,7 @@ def tpu_scaffold():
     if mode == tf.estimator.ModeKeys.TRAIN:
 
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu, use_multi_cpu=is_mpi)
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu, use_multi_cpu=is_mpi, use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, n=10)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier_with_tfhub.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier_with_tfhub.py
index 66290f5e5..2177325cc 100644
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier_with_tfhub.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_classifier_with_tfhub.py
@@ -33,6 +33,8 @@
     "bert_hub_module_handle", None,
     "Handle for the BERT TF-Hub module.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 
 def create_model(is_training, input_ids, input_mask, segment_ids, labels,
                  num_labels, bert_hub_module_handle):
@@ -109,7 +111,7 @@ def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
     output_spec = None
     if mode == tf.estimator.ModeKeys.TRAIN:
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu, use_gpu=FLAGS.gpu)
 
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
           mode=mode,
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_pretraining.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_pretraining.py
index 932983a1b..dc5d8cde4 100644
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_pretraining.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_pretraining.py
@@ -204,6 +204,7 @@ def end(self, run_context):
     "[Optional] If true, use more experimental op(gelu) in model."
     "           Be careful this flag will crash model with incompatible TF.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
 
 def model_fn_builder(bert_config, init_checkpoint, learning_rate,
                      num_train_steps, num_warmup_steps, accum_steps, use_tpu,
@@ -288,7 +289,7 @@ def tpu_scaffold():
     if mode == tf.estimator.ModeKeys.TRAIN:
       train_op = optimization.create_optimizer(
               total_loss, learning_rate, num_train_steps, num_warmup_steps, 
-                            accum_steps, use_tpu=use_tpu, use_multi_cpu=is_mpi)
+                            accum_steps, use_tpu=use_tpu, use_multi_cpu=is_mpi, use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, FLAGS.accum_steps*3)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_squad.py b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_squad.py
index c6346052a..ae86eed0c 100644
--- a/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_squad.py
+++ b/models/language_modeling/tensorflow/bert_large/training/bfloat16/run_squad.py
@@ -196,6 +196,8 @@
     "If true, turns on gradient synchronization via horovod."
     "Default is False as it gives better accuracy.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 class SquadExample(object):
   """A single training/test example for simple sequence classification.
 
@@ -714,7 +716,7 @@ def compute_loss(logits, positions):
       total_loss = (start_loss + end_loss) / 2.0
 
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=(is_mpi if FLAGS.mpi_workers_sync_gradients else 0))
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=(is_mpi if FLAGS.mpi_workers_sync_gradients else 0), use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, n=3)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/bert_large/training/fp32/optimization.py b/models/language_modeling/tensorflow/bert_large/training/fp32/optimization.py
index d8af1467e..8267da366 100644
--- a/models/language_modeling/tensorflow/bert_large/training/fp32/optimization.py
+++ b/models/language_modeling/tensorflow/bert_large/training/fp32/optimization.py
@@ -51,7 +51,7 @@
 #  show_msg(current_step, msg)
 #  return current_step.assign_add(1)
 
-def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=0):
+def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=False, use_multi_cpu=0, use_gpu=False):
   """Creates an optimizer training op."""
   global_step = tf.compat.v1.train.get_or_create_global_step()
 
@@ -87,17 +87,21 @@ def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, accum_ste
   # is how the model was trained (note that the Adam m/v variables are NOT
   # loaded from init_checkpoint.)
   use_itex_optimizer = False
-  try:
-    import intel_extension_for_tensorflow as itex
-    optimizer = itex.ops.AdamWithWeightDecayOptimizer(
-        learning_rate=learning_rate,
-        weight_decay_rate=0.01,
-        beta_1=0.9,
-        beta_2=0.999,
-        epsilon=1e-6,
-        exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
-    use_itex_optimizer = True
-  except:
+  if use_gpu:
+    try:
+      import intel_extension_for_tensorflow as itex
+      optimizer = itex.ops.AdamWithWeightDecayOptimizer(
+          learning_rate=learning_rate,
+          weight_decay_rate=0.01,
+          beta_1=0.9,
+          beta_2=0.999,
+          epsilon=1e-6,
+          exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
+      use_itex_optimizer = True
+    except:
+      print("Load plugin Intel(R) Extension for Tensorflow* failed.\n")
+
+  if not use_itex_optimizer:
     optimizer = AdamWeightDecayOptimizer(
         learning_rate=learning_rate,
         weight_decay_rate=0.01,
diff --git a/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py b/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py
index d210d451c..a3e63698e 100644
--- a/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py
+++ b/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier.py
@@ -156,6 +156,8 @@
     "[Optional] If true, use experimental gelu op in model."
     "           Be careful this flag will crash model with incompatible TF.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 
 class InputExample(object):
   """A single training/test example for simple sequence classification."""
@@ -723,7 +725,7 @@ def tpu_scaffold():
     if mode == tf.estimator.ModeKeys.TRAIN:
 
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, 1, use_tpu, is_mpi)
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, 1, use_tpu, is_mpi, use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, n=10)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier_with_tfhub.py b/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier_with_tfhub.py
index 66290f5e5..2177325cc 100644
--- a/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier_with_tfhub.py
+++ b/models/language_modeling/tensorflow/bert_large/training/fp32/run_classifier_with_tfhub.py
@@ -33,6 +33,8 @@
     "bert_hub_module_handle", None,
     "Handle for the BERT TF-Hub module.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 
 def create_model(is_training, input_ids, input_mask, segment_ids, labels,
                  num_labels, bert_hub_module_handle):
@@ -109,7 +111,7 @@ def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument
     output_spec = None
     if mode == tf.estimator.ModeKeys.TRAIN:
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu, use_gpu=FLAGS.gpu)
 
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
           mode=mode,
diff --git a/models/language_modeling/tensorflow/bert_large/training/fp32/run_pretraining.py b/models/language_modeling/tensorflow/bert_large/training/fp32/run_pretraining.py
index 7659ccdfd..94dca2036 100644
--- a/models/language_modeling/tensorflow/bert_large/training/fp32/run_pretraining.py
+++ b/models/language_modeling/tensorflow/bert_large/training/fp32/run_pretraining.py
@@ -195,6 +195,8 @@ def end(self, run_context):
     "[Optional] If true, use experimental gelu op in model."
     "           Be careful this flag will crash model with incompatible TF.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 
 def model_fn_builder(bert_config, init_checkpoint, learning_rate,
                      num_train_steps, num_warmup_steps, accum_steps, use_tpu,
@@ -265,7 +267,7 @@ def tpu_scaffold():
     output_spec = None
     if mode == tf.estimator.ModeKeys.TRAIN:
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps, use_tpu, is_mpi)
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps, use_tpu, is_mpi, use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, FLAGS.accum_steps*3)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/bert_large/training/fp32/run_squad.py b/models/language_modeling/tensorflow/bert_large/training/fp32/run_squad.py
index c9b53ec2c..eb80704c8 100644
--- a/models/language_modeling/tensorflow/bert_large/training/fp32/run_squad.py
+++ b/models/language_modeling/tensorflow/bert_large/training/fp32/run_squad.py
@@ -191,6 +191,8 @@
     "If true, turns on gradient synchronization via horovod."
     "Default is False as it gives better accuracy.")
 
+flags.DEFINE_bool("gpu", False, "Run the benchmark script using GPU")
+
 
 class SquadExample(object):
   """A single training/test example for simple sequence classification.
@@ -710,7 +712,7 @@ def compute_loss(logits, positions):
       total_loss = (start_loss + end_loss) / 2.0
 
       train_op = optimization.create_optimizer(
-          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu, use_multi_cpu=(is_mpi if FLAGS.mpi_workers_sync_gradients else 0))
+          total_loss, learning_rate, num_train_steps, num_warmup_steps, accum_steps=1, use_tpu=use_tpu, use_multi_cpu=(is_mpi if FLAGS.mpi_workers_sync_gradients else 0), use_gpu=FLAGS.gpu)
 
       log_hook = bf.logTheLossHook(total_loss, n=3)
       output_spec = tf.compat.v1.estimator.tpu.TPUEstimatorSpec(
diff --git a/models/language_modeling/tensorflow/distilbert_base/inference/bfloat16/evaluate_distilbert.py b/models/language_modeling/tensorflow/distilbert_base/inference/bfloat16/evaluate_distilbert.py
deleted file mode 100644
index b24753945..000000000
--- a/models/language_modeling/tensorflow/distilbert_base/inference/bfloat16/evaluate_distilbert.py
+++ /dev/null
@@ -1,218 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-'''Distilbert base inference, implementation adapted from Hugging Face Library https://huggingface.co/'''
-import time
-import numpy as np
-import tensorflow as tf
-from tensorflow.core.protobuf import saved_model_pb2
-from tensorflow.core.protobuf import rewriter_config_pb2
-from argparse import ArgumentParser
-from transformers import AutoTokenizer
-from datasets import load_from_disk
-
-MAX_STEPS = 872
-MAX_WARMUP_STEPS = 22
-
-class distilbert_base_inference:
-
-    def __init__(self):
-
-        arg_parser = ArgumentParser(description="Distilbert fp32 inference")
-        arg_parser.add_argument("--task-name",
-                                type=str,
-                                help= "Name of the task to run benchmark, default: sst2",
-                                default= "sst2",
-                                dest="task_name"
-                                )
-        arg_parser.add_argument("-g", "--in-graph", help="Full path to the input graph ",
-                                dest="input_graph", default=None) # TODO type=check_valid_filename)
-        arg_parser.add_argument('-e', "--num-inter-threads",
-                                help='The number of inter-thread.',
-                                dest='num_inter_threads', type=int, default=0)
-        arg_parser.add_argument('-a', "--num-intra-threads",
-                                help='The number of intra-thread.',
-                                dest='num_intra_threads', type=int, default=0)
-        arg_parser.add_argument("--pad-to-max-length",
-                                type= bool,
-                                help= "Padding option",
-                                default= True,
-                                dest= "pad_to_max_length"
-                                )
-        arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                                help="number of warmup steps",
-                                dest="warmup_steps")
-        arg_parser.add_argument("--max-seq-length", type=int, default=128,
-                                help="maximum total sequence length after Tokenization",
-                                dest="max_seq_length")
-        arg_parser.add_argument("--steps", type=int, default=850,
-                                help="number of steps")
-        arg_parser.add_argument("--data-location", type=str,
-                                help="path to saved dataset",
-                                dest="data_location")
-        arg_parser.add_argument("--batch-size",
-                                type= int,
-                                default=32,
-                                dest= "batch_size")
-        arg_parser.add_argument("--accuracy-only", action="store_true", default=None)
-        arg_parser.add_argument("--benchmark-only", action="store_true", default=None)
-        self.args = arg_parser.parse_args()
-        self.validate_args()
-
-    def validate_args(self):
-        if self.args.warmup_steps > MAX_WARMUP_STEPS:
-            print("Warmup steps greater than max possible value of 22." + \
-                  " Setting to max value of ", MAX_WARMUP_STEPS)
-            self.args.warmup_steps = MAX_WARMUP_STEPS
-        if self.args.accuracy_only:
-            self.args.steps = MAX_STEPS
-        elif self.args.benchmark_only:
-            if self.args.steps > (MAX_STEPS - MAX_WARMUP_STEPS):
-                print("Steps greater than max possible value of {}.".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                print("Setting to max value of {}".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                self.args.steps = MAX_STEPS - MAX_WARMUP_STEPS
-        if not self.args.data_location:
-            raise SystemExit("Missing dataset path")
-
-    def get_correct_predictions(self, preds, label_ids):
-        '''
-            Evaluate the predictions and return the total
-            number of correct predictions
-        '''
-        preds = np.argmax(preds, axis=1)
-        correct_preds = 0
-        for pred, label in zip(preds, label_ids):
-            if pred == label:
-                correct_preds += 1
-        return correct_preds
-
-    def preprocess_function(self, examples):
-        '''
-            Tokenize the texts
-        '''
-        sentence1_key, sentence2_key = "sentence", None
-        args = (
-            (examples[sentence1_key],) if sentence2_key is None
-            else (examples[sentence1_key], examples[sentence2_key])
-        )
-        result = self.tokenizer(*args, padding="max_length",
-                                max_length=self.args.max_seq_length,
-                                truncation=True
-        )
-        return result
-
-    def create_feed_dict_and_labels(self, batch_id= None, n_batches= None, idx= None):
-        '''
-            Return the input dictionary for the given batch
-        '''
-        if idx is None:
-            start_idx = batch_id * self.args.batch_size
-            if batch_id == n_batches - 1:
-                end_idx = self.args.steps
-            else:
-                end_idx = start_idx + self.args.batch_size
-            input_ids = np.array(self.dataset['input_ids'])[start_idx:end_idx, :]
-            attention_mask = np.array(self.dataset['attention_mask'])[start_idx:end_idx,:]
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[start_idx:end_idx]
-        else:
-            input_ids = np.array(self.dataset['input_ids'])[idx, :].reshape(1, -1)
-            attention_mask = np.array(self.dataset['attention_mask'])[idx,:].reshape(1, -1)
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[idx]
-        return feed_dict, labels
-
-    def run(self):
-        '''Run benchmark for distilbert'''
-
-        # Load dataset (only validation split for inference)
-        self.dataset = load_from_disk(self.args.data_location)
-
-        # Load tokenizer
-        self.tokenizer = AutoTokenizer.from_pretrained(
-                            "distilbert-base-uncased-finetuned-sst-2-english",
-        )
-        # Set max sequence length
-        if self.args.max_seq_length > self.tokenizer.model_max_length:
-            print(f'The max sequence length passed ({self.args.max_seq_length}) \
-                   is larger than the max supported by model \
-                   ({self.tokenizer.model_max_length}).Using max_seq_length =  \
-                   {self.tokenizer.model_max_length}')
-        self.args.max_seq_length = min(self.args.max_seq_length, self.tokenizer.model_max_length)
-
-        # Tokenize the dataset
-        self.dataset = self.dataset.map(self.preprocess_function, batched=True)
-
-        # Set the config for run
-        config = tf.compat.v1.ConfigProto()
-        config.intra_op_parallelism_threads=self.args.num_intra_threads
-        config.inter_op_parallelism_threads=self.args.num_inter_threads
-        config.graph_options.rewrite_options.auto_mixed_precision_mkl = rewriter_config_pb2.RewriterConfig.ON
-
-        # Load the frozen model
-        sm = saved_model_pb2.SavedModel()
-        with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-            sm.ParseFromString(f.read())
-        g_def = sm.meta_graphs[0].graph_def
-        with tf.Graph().as_default() as graph:
-            tf.import_graph_def(g_def, name='')
-
-        # Define the session
-        output = graph.get_tensor_by_name('Identity:0')
-        sess =  tf.compat.v1.Session(graph=graph, config=config)
-        n_batch = int(self.args.steps / self.args.batch_size)
-        if self.args.steps % self.args.batch_size != 0:
-            n_batch += 1
-
-        # Warm up
-        total_time = 0
-        print("Started warmup for {} steps...".format(self.args.warmup_steps))
-        start_step_idx = MAX_STEPS - MAX_WARMUP_STEPS
-        for step in range(start_step_idx, start_step_idx + self.args.warmup_steps):
-            feed_dict, _ = self.create_feed_dict_and_labels(idx= step)
-            _ = sess.run(output, feed_dict= feed_dict)
-        print("Warmup completed.")
-
-        # Inference
-        print("Starting inference for {} steps...".format(self.args.steps))
-        total_correct_predictions = 0
-        for batch_id in range(n_batch):
-            feed_dict, labels = self.create_feed_dict_and_labels(batch_id, n_batch)
-            start_time = time.time()
-            pred = sess.run(output, feed_dict=feed_dict)
-            run_time = time.time() - start_time
-            if self.args.accuracy_only:
-                total_correct_predictions += self.get_correct_predictions(pred, labels)
-            total_time += run_time
-        time_per_batch = total_time / float(self.args.steps / self.args.batch_size)
-
-        if self.args.accuracy_only:
-            accuracy = total_correct_predictions / self.args.steps
-            print('Accuracy: {:.4f}'.format(accuracy))
-        if self.args.benchmark_only:
-            if self.args.batch_size == 1:
-                print('Latency: {:.4f} ms'.format(time_per_batch * 1000))
-            print('Throughput: {:.4f} sentences/sec'.format(self.args.batch_size / time_per_batch))
-
-if __name__=="__main__":
-    distilbert_ob = distilbert_base_inference()
-    distilbert_ob.run()
diff --git a/models/language_modeling/tensorflow/distilbert_base/inference/fp16/evaluate_distilbert.py b/models/language_modeling/tensorflow/distilbert_base/inference/fp16/evaluate_distilbert.py
deleted file mode 100644
index 68912c1fc..000000000
--- a/models/language_modeling/tensorflow/distilbert_base/inference/fp16/evaluate_distilbert.py
+++ /dev/null
@@ -1,218 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-'''Distilbert base inference, implementation adapted from Hugging Face Library https://huggingface.co/'''
-import time
-import numpy as np
-import tensorflow as tf
-from tensorflow.core.protobuf import saved_model_pb2
-from tensorflow.core.protobuf import rewriter_config_pb2
-from argparse import ArgumentParser
-from transformers import AutoTokenizer
-from datasets import load_from_disk
-
-MAX_STEPS = 872
-MAX_WARMUP_STEPS = 22
-
-class distilbert_base_inference:
-
-    def __init__(self):
-
-        arg_parser = ArgumentParser(description="Distilbert fp32 inference")
-        arg_parser.add_argument("--task-name",
-                                type=str,
-                                help= "Name of the task to run benchmark, default: sst2",
-                                default= "sst2",
-                                dest="task_name"
-                                )
-        arg_parser.add_argument("-g", "--in-graph", help="Full path to the input graph ",
-                                dest="input_graph", default=None) # TODO type=check_valid_filename)
-        arg_parser.add_argument('-e', "--num-inter-threads",
-                                help='The number of inter-thread.',
-                                dest='num_inter_threads', type=int, default=0)
-        arg_parser.add_argument('-a', "--num-intra-threads",
-                                help='The number of intra-thread.',
-                                dest='num_intra_threads', type=int, default=0)
-        arg_parser.add_argument("--pad-to-max-length",
-                                type= bool,
-                                help= "Padding option",
-                                default= True,
-                                dest= "pad_to_max_length"
-                                )
-        arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                                help="number of warmup steps",
-                                dest="warmup_steps")
-        arg_parser.add_argument("--max-seq-length", type=int, default=128,
-                                help="maximum total sequence length after Tokenization",
-                                dest="max_seq_length")
-        arg_parser.add_argument("--steps", type=int, default=850,
-                                help="number of steps")
-        arg_parser.add_argument("--data-location", type=str,
-                                help="path to saved dataset",
-                                dest="data_location")
-        arg_parser.add_argument("--batch-size",
-                                type= int,
-                                default=32,
-                                dest= "batch_size")
-        arg_parser.add_argument("--accuracy-only", action="store_true", default=None)
-        arg_parser.add_argument("--benchmark-only", action="store_true", default=None)
-        self.args = arg_parser.parse_args()
-        self.validate_args()
-
-    def validate_args(self):
-        if self.args.warmup_steps > MAX_WARMUP_STEPS:
-            print("Warmup steps greater than max possible value of 22." + \
-                  " Setting to max value of ", MAX_WARMUP_STEPS)
-            self.args.warmup_steps = MAX_WARMUP_STEPS
-        if self.args.accuracy_only:
-            self.args.steps = MAX_STEPS
-        elif self.args.benchmark_only:
-            if self.args.steps > (MAX_STEPS - MAX_WARMUP_STEPS):
-                print("Steps greater than max possible value of {}.".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                print("Setting to max value of {}".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                self.args.steps = MAX_STEPS - MAX_WARMUP_STEPS
-        if not self.args.data_location:
-            raise SystemExit("Missing dataset path")
-
-    def get_correct_predictions(self, preds, label_ids):
-        '''
-            Evaluate the predictions and return the total
-            number of correct predictions
-        '''
-        preds = np.argmax(preds, axis=1)
-        correct_preds = 0
-        for pred, label in zip(preds, label_ids):
-            if pred == label:
-                correct_preds += 1
-        return correct_preds
-
-    def preprocess_function(self, examples):
-        '''
-            Tokenize the texts
-        '''
-        sentence1_key, sentence2_key = "sentence", None
-        args = (
-            (examples[sentence1_key],) if sentence2_key is None
-            else (examples[sentence1_key], examples[sentence2_key])
-        )
-        result = self.tokenizer(*args, padding="max_length",
-                                max_length=self.args.max_seq_length,
-                                truncation=True
-        )
-        return result
-
-    def create_feed_dict_and_labels(self, batch_id= None, n_batches= None, idx= None):
-        '''
-            Return the input dictionary for the given batch
-        '''
-        if idx is None:
-            start_idx = batch_id * self.args.batch_size
-            if batch_id == n_batches - 1:
-                end_idx = self.args.steps
-            else:
-                end_idx = start_idx + self.args.batch_size
-            input_ids = np.array(self.dataset['input_ids'])[start_idx:end_idx, :]
-            attention_mask = np.array(self.dataset['attention_mask'])[start_idx:end_idx,:]
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[start_idx:end_idx]
-        else:
-            input_ids = np.array(self.dataset['input_ids'])[idx, :].reshape(1, -1)
-            attention_mask = np.array(self.dataset['attention_mask'])[idx,:].reshape(1, -1)
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[idx]
-        return feed_dict, labels
-
-    def run(self):
-        '''Run benchmark for distilbert'''
-
-        # Load dataset (only validation split for inference)
-        self.dataset = load_from_disk(self.args.data_location)
-
-        # Load tokenizer
-        self.tokenizer = AutoTokenizer.from_pretrained(
-                            "distilbert-base-uncased-finetuned-sst-2-english",
-        )
-        # Set max sequence length
-        if self.args.max_seq_length > self.tokenizer.model_max_length:
-            print(f'The max sequence length passed ({self.args.max_seq_length}) \
-                   is larger than the max supported by model \
-                   ({self.tokenizer.model_max_length}).Using max_seq_length =  \
-                   {self.tokenizer.model_max_length}')
-        self.args.max_seq_length = min(self.args.max_seq_length, self.tokenizer.model_max_length)
-
-        # Tokenize the dataset
-        self.dataset = self.dataset.map(self.preprocess_function, batched=True)
-
-        # Set the config for run
-        config = tf.compat.v1.ConfigProto()
-        config.intra_op_parallelism_threads=self.args.num_intra_threads
-        config.inter_op_parallelism_threads=self.args.num_inter_threads
-        config.graph_options.rewrite_options.auto_mixed_precision = rewriter_config_pb2.RewriterConfig.ON
-
-        # Load the frozen model
-        sm = saved_model_pb2.SavedModel()
-        with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-            sm.ParseFromString(f.read())
-        g_def = sm.meta_graphs[0].graph_def
-        with tf.Graph().as_default() as graph:
-            tf.import_graph_def(g_def, name='')
-
-        # Define the session
-        output = graph.get_tensor_by_name('Identity:0')
-        sess =  tf.compat.v1.Session(graph=graph, config=config)
-        n_batch = int(self.args.steps / self.args.batch_size)
-        if self.args.steps % self.args.batch_size != 0:
-            n_batch += 1
-
-        # Warm up
-        total_time = 0
-        print("Started warmup for {} steps...".format(self.args.warmup_steps))
-        start_step_idx = MAX_STEPS - MAX_WARMUP_STEPS
-        for step in range(start_step_idx, start_step_idx + self.args.warmup_steps):
-            feed_dict, _ = self.create_feed_dict_and_labels(idx= step)
-            _ = sess.run(output, feed_dict= feed_dict)
-        print("Warmup completed.")
-
-        # Inference
-        print("Starting inference for {} steps...".format(self.args.steps))
-        total_correct_predictions = 0
-        for batch_id in range(n_batch):
-            feed_dict, labels = self.create_feed_dict_and_labels(batch_id, n_batch)
-            start_time = time.time()
-            pred = sess.run(output, feed_dict=feed_dict)
-            run_time = time.time() - start_time
-            if self.args.accuracy_only:
-                total_correct_predictions += self.get_correct_predictions(pred, labels)
-            total_time += run_time
-        time_per_batch = total_time / float(self.args.steps / self.args.batch_size)
-
-        if self.args.accuracy_only:
-            accuracy = total_correct_predictions / self.args.steps
-            print('Accuracy: {:.4f}'.format(accuracy))
-        if self.args.benchmark_only:
-            if self.args.batch_size == 1:
-                print('Latency: {:.4f} ms'.format(time_per_batch * 1000))
-            print('Throughput: {:.4f} sentences/sec'.format(self.args.batch_size / time_per_batch))
-
-if __name__=="__main__":
-    distilbert_ob = distilbert_base_inference()
-    distilbert_ob.run()
diff --git a/models/language_modeling/tensorflow/distilbert_base/inference/fp32/evaluate_distilbert.py b/models/language_modeling/tensorflow/distilbert_base/inference/fp32/evaluate_distilbert.py
deleted file mode 100644
index 126f160e3..000000000
--- a/models/language_modeling/tensorflow/distilbert_base/inference/fp32/evaluate_distilbert.py
+++ /dev/null
@@ -1,216 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-'''Distilbert base inference, implementation adapted from Hugging Face Library https://huggingface.co/'''
-import time
-import numpy as np
-import tensorflow as tf
-from tensorflow.core.protobuf import saved_model_pb2
-from argparse import ArgumentParser
-from transformers import AutoTokenizer
-from datasets import load_from_disk
-
-MAX_STEPS = 872
-MAX_WARMUP_STEPS = 22
-
-class distilbert_base_inference:
-
-    def __init__(self):
-
-        arg_parser = ArgumentParser(description="Distilbert fp32 inference")
-        arg_parser.add_argument("--task-name",
-                                type=str,
-                                help= "Name of the task to run benchmark, default: sst2",
-                                default= "sst2",
-                                dest="task_name"
-                                )
-        arg_parser.add_argument("-g", "--in-graph", help="Full path to the input graph ",
-                                dest="input_graph", default=None) # TODO type=check_valid_filename)
-        arg_parser.add_argument('-e', "--num-inter-threads",
-                                help='The number of inter-thread.',
-                                dest='num_inter_threads', type=int, default=0)
-        arg_parser.add_argument('-a', "--num-intra-threads",
-                                help='The number of intra-thread.',
-                                dest='num_intra_threads', type=int, default=0)
-        arg_parser.add_argument("--pad-to-max-length",
-                                type= bool,
-                                help= "Padding option",
-                                default= True,
-                                dest= "pad_to_max_length"
-                                )
-        arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                                help="number of warmup steps",
-                                dest="warmup_steps")
-        arg_parser.add_argument("--max-seq-length", type=int, default=128,
-                                help="maximum total sequence length after Tokenization",
-                                dest="max_seq_length")
-        arg_parser.add_argument("--steps", type=int, default=850,
-                                help="number of steps")
-        arg_parser.add_argument("--data-location", type=str,
-                                help="path to saved dataset",
-                                dest="data_location")
-        arg_parser.add_argument("--batch-size",
-                                type= int,
-                                default=32,
-                                dest= "batch_size")
-        arg_parser.add_argument("--accuracy-only", action="store_true", default=None)
-        arg_parser.add_argument("--benchmark-only", action="store_true", default=None)
-        self.args = arg_parser.parse_args()
-        self.validate_args()
-
-    def validate_args(self):
-        if self.args.warmup_steps > MAX_WARMUP_STEPS:
-            print("Warmup steps greater than max possible value of 22." + \
-                  " Setting to max value of ", MAX_WARMUP_STEPS)
-            self.args.warmup_steps = MAX_WARMUP_STEPS
-        if self.args.accuracy_only:
-            self.args.steps = MAX_STEPS
-        elif self.args.benchmark_only:
-            if self.args.steps > (MAX_STEPS - MAX_WARMUP_STEPS):
-                print("Steps greater than max possible value of {}.".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                print("Setting to max value of {}".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                self.args.steps = MAX_STEPS - MAX_WARMUP_STEPS
-        if not self.args.data_location:
-            raise SystemExit("Missing dataset path")
-
-    def get_correct_predictions(self, preds, label_ids):
-        '''
-            Evaluate the predictions and return the total
-            number of correct predictions
-        '''
-        preds = np.argmax(preds, axis=1)
-        correct_preds = 0
-        for pred, label in zip(preds, label_ids):
-            if pred == label:
-                correct_preds += 1
-        return correct_preds
-
-    def preprocess_function(self, examples):
-        '''
-            Tokenize the texts
-        '''
-        sentence1_key, sentence2_key = "sentence", None
-        args = (
-            (examples[sentence1_key],) if sentence2_key is None
-            else (examples[sentence1_key], examples[sentence2_key])
-        )
-        result = self.tokenizer(*args, padding="max_length",
-                                max_length=self.args.max_seq_length,
-                                truncation=True
-        )
-        return result
-
-    def create_feed_dict_and_labels(self, batch_id= None, n_batches= None, idx= None):
-        '''
-            Return the input dictionary for the given batch
-        '''
-        if idx is None:
-            start_idx = batch_id * self.args.batch_size
-            if batch_id == n_batches - 1:
-                end_idx = self.args.steps
-            else:
-                end_idx = start_idx + self.args.batch_size
-            input_ids = np.array(self.dataset['input_ids'])[start_idx:end_idx, :]
-            attention_mask = np.array(self.dataset['attention_mask'])[start_idx:end_idx,:]
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[start_idx:end_idx]
-        else:
-            input_ids = np.array(self.dataset['input_ids'])[idx, :].reshape(1, -1)
-            attention_mask = np.array(self.dataset['attention_mask'])[idx,:].reshape(1, -1)
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[idx]
-        return feed_dict, labels
-
-    def run(self):
-        '''Run benchmark for distilbert'''
-
-        # Load dataset (only validation split for inference)
-        self.dataset = load_from_disk(self.args.data_location)
-
-        # Load tokenizer
-        self.tokenizer = AutoTokenizer.from_pretrained(
-                            "distilbert-base-uncased-finetuned-sst-2-english",
-        )
-        # Set max sequence length
-        if self.args.max_seq_length > self.tokenizer.model_max_length:
-            print(f'The max sequence length passed ({self.args.max_seq_length}) \
-                   is larger than the max supported by model \
-                   ({self.tokenizer.model_max_length}).Using max_seq_length =  \
-                   {self.tokenizer.model_max_length}')
-        self.args.max_seq_length = min(self.args.max_seq_length, self.tokenizer.model_max_length)
-
-        # Tokenize the dataset
-        self.dataset = self.dataset.map(self.preprocess_function, batched=True)
-
-        # Set the config for run
-        config = tf.compat.v1.ConfigProto()
-        config.intra_op_parallelism_threads=self.args.num_intra_threads
-        config.inter_op_parallelism_threads=self.args.num_inter_threads
-
-        # Load the frozen model
-        sm = saved_model_pb2.SavedModel()
-        with tf.io.gfile.GFile(self.args.input_graph, "rb") as f:
-            sm.ParseFromString(f.read())
-        g_def = sm.meta_graphs[0].graph_def
-        with tf.Graph().as_default() as graph:
-            tf.import_graph_def(g_def, name='')
-
-        # Define the session
-        output = graph.get_tensor_by_name('Identity:0')
-        sess =  tf.compat.v1.Session(graph=graph, config=config)
-        n_batch = int(self.args.steps / self.args.batch_size)
-        if self.args.steps % self.args.batch_size != 0:
-            n_batch += 1
-
-        # Warm up
-        total_time = 0
-        print("Started warmup for {} steps...".format(self.args.warmup_steps))
-        start_step_idx = MAX_STEPS - MAX_WARMUP_STEPS
-        for step in range(start_step_idx, start_step_idx + self.args.warmup_steps):
-            feed_dict, _ = self.create_feed_dict_and_labels(idx= step)
-            _ = sess.run(output, feed_dict= feed_dict)
-        print("Warmup completed.")
-
-        # Inference
-        print("Starting inference for {} steps...".format(self.args.steps))
-        total_correct_predictions = 0
-        for batch_id in range(n_batch):
-            feed_dict, labels = self.create_feed_dict_and_labels(batch_id, n_batch)
-            start_time = time.time()
-            pred = sess.run(output, feed_dict=feed_dict)
-            run_time = time.time() - start_time
-            if self.args.accuracy_only:
-                total_correct_predictions += self.get_correct_predictions(pred, labels)
-            total_time += run_time
-        time_per_batch = total_time / float(self.args.steps / self.args.batch_size)
-
-        if self.args.accuracy_only:
-            accuracy = total_correct_predictions / self.args.steps
-            print('Accuracy: {:.4f}'.format(accuracy))
-        if self.args.benchmark_only:
-            if self.args.batch_size == 1:
-                print('Latency: {:.4f} ms'.format(time_per_batch * 1000))
-            print('Throughput: {:.4f} sentences/sec'.format(self.args.batch_size / time_per_batch))
-
-if __name__=="__main__":
-    distilbert_ob = distilbert_base_inference()
-    distilbert_ob.run()
diff --git a/models/language_modeling/tensorflow/distilbert_base/inference/int8/evaluate_distilbert.py b/models/language_modeling/tensorflow/distilbert_base/inference/int8/evaluate_distilbert.py
deleted file mode 100644
index 24c04dd57..000000000
--- a/models/language_modeling/tensorflow/distilbert_base/inference/int8/evaluate_distilbert.py
+++ /dev/null
@@ -1,216 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-'''Distilbert base inference, implementation adapted from Hugging Face Library https://huggingface.co/'''
-import time, sys
-import numpy as np
-import tensorflow as tf
-from tensorflow.core.protobuf import saved_model_pb2
-from argparse import ArgumentParser
-from transformers import AutoTokenizer
-from datasets import load_from_disk
-
-MAX_STEPS = 872
-MAX_WARMUP_STEPS = 22
-
-class distilbert_base_inference:
-
-    def __init__(self):
-
-        arg_parser = ArgumentParser(description="Distilbert int8 inference")
-        arg_parser.add_argument("--task-name",
-                                type=str,
-                                help= "Name of the task to run benchmark, default: sst2",
-                                default= "sst2",
-                                dest="task_name"
-                                )
-        arg_parser.add_argument("-g", "--in-graph", help="Full path to the input graph ",
-                                dest="input_graph", default=None) # TODO type=check_valid_filename)
-        arg_parser.add_argument('-e', "--num-inter-threads",
-                                help='The number of inter-thread.',
-                                dest='num_inter_threads', type=int, default=0)
-        arg_parser.add_argument('-a', "--num-intra-threads",
-                                help='The number of intra-thread.',
-                                dest='num_intra_threads', type=int, default=0)
-        arg_parser.add_argument("--pad-to-max-length",
-                                type= bool,
-                                help= "Padding option",
-                                default= True,
-                                dest= "pad_to_max_length"
-                                )
-        arg_parser.add_argument("--warmup-steps", type=int, default=10,
-                                help="number of warmup steps",
-                                dest="warmup_steps")
-        arg_parser.add_argument("--max-seq-length", type=int, default=128,
-                                help="maximum total sequence length after Tokenization",
-                                dest="max_seq_length")
-        arg_parser.add_argument("--steps", type=int, default=850,
-                                help="number of steps")
-        arg_parser.add_argument("--data-location", type=str,
-                                help="path to saved dataset",
-                                dest="data_location")
-        arg_parser.add_argument("--batch-size",
-                                type= int,
-                                default=32,
-                                dest= "batch_size")
-        arg_parser.add_argument("--accuracy-only", action="store_true", default=None)
-        arg_parser.add_argument("--benchmark-only", action="store_true", default=None)
-        self.args = arg_parser.parse_args()
-        self.validate_args()
-
-    def validate_args(self):
-        if self.args.warmup_steps > MAX_WARMUP_STEPS:
-            print("Warmup steps greater than max possible value of 22." + \
-                  " Setting to max value of ", MAX_WARMUP_STEPS)
-            self.args.warmup_steps = MAX_WARMUP_STEPS
-        if self.args.accuracy_only:
-            self.args.steps = MAX_STEPS
-        elif self.args.benchmark_only:
-            if self.args.steps > (MAX_STEPS - MAX_WARMUP_STEPS):
-                print("Steps greater than max possible value of {}.".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                print("Setting to max value of {}".format(MAX_STEPS - MAX_WARMUP_STEPS))
-                self.args.steps = MAX_STEPS - MAX_WARMUP_STEPS
-        if not self.args.data_location:
-            raise SystemExit("Missing dataset path")
-
-    def get_correct_predictions(self, preds, label_ids):
-        '''
-            Evaluate the predictions and return the total
-            number of correct predictions
-        '''
-        preds = np.argmax(preds, axis=1)
-        correct_preds = 0
-        for pred, label in zip(preds, label_ids):
-            if pred == label:
-                correct_preds += 1
-        return correct_preds
-
-    def preprocess_function(self, examples):
-        '''
-            Tokenize the texts
-        '''
-        sentence1_key, sentence2_key = "sentence", None
-        args = (
-            (examples[sentence1_key],) if sentence2_key is None
-            else (examples[sentence1_key], examples[sentence2_key])
-        )
-        result = self.tokenizer(*args, padding="max_length",
-                                max_length=self.args.max_seq_length,
-                                truncation=True
-        )
-        return result
-
-    def create_feed_dict_and_labels(self, batch_id= None, n_batches= None, idx= None):
-        '''
-            Return the input dictionary for the given batch
-        '''
-        if idx is None:
-            start_idx = batch_id * self.args.batch_size
-            if batch_id == n_batches - 1:
-                end_idx = self.args.steps
-            else:
-                end_idx = start_idx + self.args.batch_size
-            input_ids = np.array(self.dataset['input_ids'])[start_idx:end_idx, :]
-            attention_mask = np.array(self.dataset['attention_mask'])[start_idx:end_idx,:]
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[start_idx:end_idx]
-        else:
-            input_ids = np.array(self.dataset['input_ids'])[idx, :].reshape(1, -1)
-            attention_mask = np.array(self.dataset['attention_mask'])[idx,:].reshape(1, -1)
-            feed_dict = { 'input_ids:0': input_ids,
-                        'attention_mask:0': attention_mask,
-            }
-            labels = np.array(self.dataset["label"])[idx]
-        return feed_dict, labels
-
-    def run(self):
-        '''Run benchmark for distilbert'''
-        print("Run INT8 inference")
-        # Load dataset (only validation split for inference)
-        self.dataset = load_from_disk(self.args.data_location)
-
-        # Load tokenizer
-        self.tokenizer = AutoTokenizer.from_pretrained(
-                            "distilbert-base-uncased-finetuned-sst-2-english",
-        )
-        # Set max sequence length
-        if self.args.max_seq_length > self.tokenizer.model_max_length:
-            print(f'The max sequence length passed ({self.args.max_seq_length}) \
-                   is larger than the max supported by model \
-                   ({self.tokenizer.model_max_length}).Using max_seq_length =  \
-                   {self.tokenizer.model_max_length}')
-        self.args.max_seq_length = min(self.args.max_seq_length, self.tokenizer.model_max_length)
-
-        # Tokenize the dataset
-        self.dataset = self.dataset.map(self.preprocess_function, batched=True)
-
-        # Set the config for run
-        config = tf.compat.v1.ConfigProto()
-        config.intra_op_parallelism_threads=self.args.num_intra_threads
-        config.inter_op_parallelism_threads=self.args.num_inter_threads
-
-        # Load the frozen model
-        graph = tf.Graph()
-        with graph.as_default():
-            graph_def = tf.compat.v1.GraphDef()
-            with open(self.args.input_graph, "rb") as f:
-                graph_def.ParseFromString(f.read())
-            tf.import_graph_def(graph_def, name="")
-
-        # Define the session
-        output = graph.get_tensor_by_name('Identity:0')
-        sess =  tf.compat.v1.Session(graph=graph, config=config)
-        n_batch = int(self.args.steps / self.args.batch_size)
-        if self.args.steps % self.args.batch_size != 0:
-            n_batch += 1
-
-        # Warm up
-        total_time = 0
-        print("Started warmup for {} steps...".format(self.args.warmup_steps))
-        start_step_idx = MAX_STEPS - MAX_WARMUP_STEPS
-        for step in range(start_step_idx, start_step_idx + self.args.warmup_steps):
-            feed_dict, _ = self.create_feed_dict_and_labels(idx= step)
-            _ = sess.run(output, feed_dict= feed_dict)
-        print("Warmup completed.")
-
-        # Inference
-        print("Starting inference for {} steps...".format(self.args.steps))
-        total_correct_predictions = 0
-        for batch_id in range(n_batch):
-            feed_dict, labels = self.create_feed_dict_and_labels(batch_id, n_batch)
-            start_time = time.time()
-            pred = sess.run(output, feed_dict=feed_dict)
-            run_time = time.time() - start_time
-            if self.args.accuracy_only:
-                total_correct_predictions += self.get_correct_predictions(pred, labels)
-            total_time += run_time
-        time_per_batch = total_time / float(self.args.steps / self.args.batch_size)
-
-        if self.args.accuracy_only:
-            accuracy = total_correct_predictions / self.args.steps
-            print('Accuracy: {:.4f}'.format(accuracy))
-        if self.args.benchmark_only:
-            if self.args.batch_size == 1:
-                print('Latency: {:.4f} ms'.format(time_per_batch * 1000))
-            print('Throughput: {:.4f} sentences/sec'.format(self.args.batch_size / time_per_batch))
-
-if __name__=="__main__":
-    distilbert_ob = distilbert_base_inference()
-    distilbert_ob.run()
diff --git a/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/attention_layer.py b/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/attention_layer.py
index 6a48eef0a..4b1a02ada 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/attention_layer.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/attention_layer.py
@@ -131,20 +131,22 @@ def call(self, x, y, bias, cache=None, encdec_cache=None):
         else:
           k = self.k_dense_layer(y)
           v = self.v_dense_layer(y)
-
+    # Split q, k, v into heads.
+    # Only perform split_heads on k, v if not dealing with k, v encoder-decoder caches
+    q = self.split_heads(q)
+    if not encdec_cache:
+      k = self.split_heads(k)
+      v = self.split_heads(v)
+      
     if cache is not None:
       # Combine cached keys and values with new keys and values.
-      k = tf.concat([cache["k"], k], axis=1)
-      v = tf.concat([cache["v"], v], axis=1)
+      k = tf.concat([cache["k"], k], axis=2)
+      v = tf.concat([cache["v"], v], axis=2)
 
       # Update cache
       cache["k"] = k
       cache["v"] = v
 
-    # Split q, k, v into heads.
-    q = self.split_heads(q)
-    k = self.split_heads(k)
-    v = self.split_heads(v)
 
     # Scale q to prevent the dot product between q and k from growing too large.
     # Calculate dot product attention
diff --git a/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/transformer.py b/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/transformer.py
index 7d51e410f..c767e4b07 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/transformer.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/inference/bfloat16/transformer/model/transformer.py
@@ -230,9 +230,9 @@ def predict(self, encoder_outputs, encoder_decoder_attention_bias):
     # Create cache storing decoder attention values for each layer.
     cache = {
         "layer_%d" % layer: {
-            "k": tf.zeros([batch_size, 0, self.params.hidden_size],
+            "k": tf.zeros([batch_size, self.params.num_heads, 0, self.params.hidden_size//self.params.num_heads],
                            dtype=tf.bfloat16),
-            "v": tf.zeros([batch_size, 0, self.params.hidden_size],
+            "v": tf.zeros([batch_size, self.params.num_heads, 0, self.params.hidden_size//self.params.num_heads],
                            dtype=tf.bfloat16)
         } for layer in range(self.params.num_hidden_layers)}
 
@@ -383,6 +383,8 @@ def __init__(self, params, train):
     self.batch_size = params.batch_size
     self.beam_size = params.beam_size
     self.hidden_size = params.hidden_size
+    self.enc_dec_cache_helper = attention_layer.Attention(
+          params.hidden_size, params.num_heads, params.attention_dropout, train)
     mlperf_log.transformer_print(
         key=mlperf_log.MODEL_HP_NUM_HIDDEN_LAYERS,
         value=params.num_hidden_layers)
@@ -428,11 +430,8 @@ def call(self, decoder_inputs, encoder_outputs, decoder_self_attention_bias,
 
   def cache_encdec(self, encoder_outputs, encdec_attention_bias):
     with tf.compat.v1.name_scope("encdec_cache"):
-      encoder_outputs = beam_search._expand_to_beam_size(encoder_outputs, self.beam_size)
-      encoder_outputs = beam_search._flatten_beam_dim(encoder_outputs)
       encdec_attention_bias = beam_search._expand_to_beam_size(encdec_attention_bias, self.beam_size)
       encdec_attention_bias = beam_search._flatten_beam_dim(encdec_attention_bias)
-      self.enc_out_cache["encoder_outputs"] = encoder_outputs
       self.enc_out_cache["encoder_decoder_attention_bias"] = encdec_attention_bias
     for n, layer in enumerate(self.layers):
       with tf.compat.v1.variable_scope("decoder_stack/" + "layer_%d" % n + "/encdec_attention", reuse=tf.compat.v1.AUTO_REUSE):
@@ -442,5 +441,17 @@ def cache_encdec(self, encoder_outputs, encdec_attention_bias):
           shape=[self.hidden_size, self.hidden_size])
         with tf.compat.v1.tpu.bfloat16_scope("attention/"):
           k = tf.matmul(encoder_outputs, tf.cast(Wk, tf.bfloat16))
+          k = beam_search._expand_to_beam_size(k, self.beam_size)
+          k = beam_search._flatten_beam_dim(k)
+          k = self.enc_dec_cache_helper.split_heads(k)
+          
           v = tf.matmul(encoder_outputs, tf.cast(Wv, tf.bfloat16))
+          v = beam_search._expand_to_beam_size(v, self.beam_size) 
+          v = beam_search._flatten_beam_dim(v)
+          v = self.enc_dec_cache_helper.split_heads(v)
           self.encdec_cache["layer_%d" % n] = {"k": k, "v": v}
+    with tf.compat.v1.name_scope("encdec_cache"):
+      encoder_outputs = beam_search._expand_to_beam_size(encoder_outputs, self.beam_size)
+      encoder_outputs = beam_search._flatten_beam_dim(encoder_outputs)
+      self.enc_out_cache["encoder_outputs"] = encoder_outputs
+      
diff --git a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/model/model_params.py b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/model/model_params.py
index 5a27390a7..e3e2eab4e 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/model/model_params.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/model/model_params.py
@@ -51,6 +51,8 @@ class TransformerBaseParams(object):
   beam_size = 4
   alpha = 0.6  # used to calculate length normalization in beam search
 
+  #  Other params
+  onednn_graph = False
 
 class TransformerBigParams(TransformerBaseParams):
   """Parameters for the big Transformer model."""
diff --git a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/transformer_main.py b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/transformer_main.py
index 888ba25be..f8c9b018c 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/transformer_main.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/transformer_main.py
@@ -56,7 +56,10 @@ def model_fn(features, labels, mode, params):
   if params.frozen_graph and mode == tf.estimator.ModeKeys.PREDICT:
     print("Reading***** From *** pb", flush=True)
     input_map = {'input_tokens': features}
-    output_names = ['model/Transformer/strided_slice_15',
+    if params.onednn_graph:
+      output_names = ['model/Transformer/strided_slice_15']
+    else:
+      output_names = ['model/Transformer/strided_slice_15',
                     'model/Transformer/strided_slice_16']
 
     with tf.io.gfile.GFile(params.frozen_graph, "rb") as f:
@@ -64,7 +67,10 @@ def model_fn(features, labels, mode, params):
       graph_def.ParseFromString(f.read())
     tf.graph_util.import_graph_def(graph_def, input_map, output_names, name="")
     output_tensors = [tf.compat.v1.get_default_graph().get_tensor_by_name(name + ":0") for name in output_names]
-    output = {'outputs': output_tensors[0] , 'scores': output_tensors[1]}
+    if params.onednn_graph:
+      output = {'outputs': output_tensors[0]}
+    else:
+      output = {'outputs': output_tensors[0] , 'scores': output_tensors[1]}
     return tf.estimator.EstimatorSpec(
         tf.estimator.ModeKeys.PREDICT,
         predictions=output)
@@ -409,6 +415,7 @@ def main(_):
   params.num_cpu_cores = FLAGS.num_cpu_cores
   params.epochs_between_eval = FLAGS.epochs_between_eval
   params.repeat_dataset = single_iteration_train_epochs
+  params.onednn_graph = FLAGS.onednn_graph
   # Add inter_op and intra_op parallelism thread
   session_config = tf.compat.v1.ConfigProto(
       inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
@@ -532,6 +539,10 @@ def main(_):
   parser.add_argument(
       "--print_iter", "-pi", type=int, default=None,
       help="print_iteration to print loss and timing", metavar="<PI>")
+  parser.add_argument(
+      "--onednn-graph", dest="--onednn_graph",
+      help="enable onednn graph for itex users",
+      action='store_true')
 
   FLAGS, unparsed = parser.parse_known_args()
 
diff --git a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/translate.py b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/translate.py
index 48e421b1d..3549dbf66 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/translate.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/inference/int8/transformer/translate.py
@@ -29,6 +29,7 @@
 from tensorflow.python.training import training_util
 from tensorflow.python.training import session_run_hook
 from tensorflow.python.training.basic_session_run_hooks import StopAtStepHook, ProfilerHook
+from tensorflow.core.protobuf import rewriter_config_pb2
 
 from mlperf_compliance import mlperf_log
 
@@ -223,6 +224,24 @@ def input_fn():
 def main(unused_argv):
   tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
 
+  if FLAGS.onednn_graph:
+    import intel_extension_for_tensorflow as itex
+    auto_mixed_precision_options = itex.AutoMixedPrecisionOptions()
+    auto_mixed_precision_options.data_type = itex.BFLOAT16
+
+    auto_mixed_precision_options.allowlist_add= "MatMul,Mul,Sub,_MklLayerNorm,ExpandDims"
+    auto_mixed_precision_options.inferlist_remove = "MatMul,Mul,Sub,_MklLayerNorm"
+    auto_mixed_precision_options.clearlist_remove = "ExpandDims"
+
+    graph_options = itex.GraphOptions(auto_mixed_precision_options=auto_mixed_precision_options)
+    graph_options.auto_mixed_precision = itex.ON
+
+    config = itex.ConfigProto(graph_options=graph_options)
+    try:
+      itex.set_backend("cpu", config)
+    except Exception:
+      itex.set_config(config)
+
   if FLAGS.text == None and FLAGS.file == None:
     tf.compat.v1.logging.warn("Nothing to translate. Make sure to call this script using "
                     "flags --text or --file.")
@@ -245,12 +264,16 @@ def main(unused_argv):
   params.extra_decode_length = _EXTRA_DECODE_LENGTH
   params.batch_size = FLAGS.batch_size
   params.frozen_graph = FLAGS.input_graph
+  params.onednn_graph = FLAGS.onednn_graph
 
   # Add inter_op and intra_op parallelism thread
   session_config = tf.compat.v1.ConfigProto(
       inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
       intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads,
       allow_soft_placement=True)
+  if FLAGS.onednn_graph:
+    session_config.graph_options.rewrite_options.constant_folding = (
+      rewriter_config_pb2.RewriterConfig.OFF)
   run_config = tf.estimator.RunConfig(session_config=session_config)
   estimator = tf.estimator.Estimator(
       model_fn=transformer_main.model_fn, model_dir=FLAGS.model_dir, params=params,
@@ -336,6 +359,10 @@ def main(unused_argv):
       help="Number of steps for benchmark.")
   parser.add_argument("--input_graph", type=str, default=None,
       help="Frozen graph path.")
+  parser.add_argument(
+      "--onednn-graph", dest="onednn_graph",
+      help="enable onednn graph for itex users",
+      action='store_true')
 
   FLAGS, unparsed = parser.parse_known_args()
   main(sys.argv)
diff --git a/models/language_translation/tensorflow/transformer_mlperf/training/bfloat16/transformer/transformer_main.py b/models/language_translation/tensorflow/transformer_mlperf/training/bfloat16/transformer/transformer_main.py
index fd7dd482d..019145f39 100644
--- a/models/language_translation/tensorflow/transformer_mlperf/training/bfloat16/transformer/transformer_main.py
+++ b/models/language_translation/tensorflow/transformer_mlperf/training/bfloat16/transformer/transformer_main.py
@@ -156,7 +156,7 @@ def get_train_op(loss, params):
     global_step = tf.compat.v1.train.get_global_step()
     tvars = tf.compat.v1.trainable_variables()
     grads_and_vars = optimizer.compute_gradients(
-        loss, tvars)
+        loss, tvars, gate_gradients=tf.compat.v1.train.Optimizer.GATE_NONE)
     train_op = optimizer.apply_gradients(
         grads_and_vars,global_step=global_step, name="train")
     # Save gradient norm to Tensorboard
@@ -320,7 +320,7 @@ def train_schedule(
   mlperf_log.transformer_print(key=mlperf_log.TRAIN_LOOP)
   # Profiling with timeline
   if FLAGS.save_profile == "Yes":
-    profile_hooks = [tf.compat.v1.train.ProfilerHook(save_steps=1, output_dir=FLAGS.profile_dir)] # the json file 
+    profile_hooks = [tf.compat.v1.train.ProfilerHook(save_steps=1, output_dir=FLAGS.profile_dir)] # the json file
   #profile file will be saved in in profile_dir
   #Creating hooks for printing Examples per Second, used with estimator.train
   training_batch_size = estimator.params.batch_size
diff --git a/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py b/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py
index 4efdf4696..e85658ecf 100644
--- a/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py
+++ b/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py
@@ -483,17 +483,6 @@ def train300_mlperf_coco(args):
     else:
         print("training with fp32 data type")
 
-    if args.bf32 and args.autocast:
-        assert False, "args.bf32 and args.autocast can't be set together"
-        exit(-1)
-
-    if args.autocast:
-        print("training with bf16 data type autocast")
-    elif args.bf32:
-        print("training with bf32 fpmath mode")
-    else:
-        print("training with fp32 data type")
-
     optim.zero_grad(set_to_none=True)
     for epoch in range(args.epochs):
         mllogger.start(
diff --git a/models/object_detection/pytorch/yolov4/inference/gpu/data/000000581918.jpg b/models/object_detection/pytorch/yolov4/inference/gpu/data/000000581918.jpg
deleted file mode 100644
index ba93d5310..000000000
Binary files a/models/object_detection/pytorch/yolov4/inference/gpu/data/000000581918.jpg and /dev/null differ
diff --git a/models/object_detection/tensorflow/ssd-mobilenet/inference/gpu/int8/infer_detections.py b/models/object_detection/tensorflow/ssd-mobilenet/inference/gpu/int8/infer_detections.py
index 440fe52b1..3e77f66f5 100644
--- a/models/object_detection/tensorflow/ssd-mobilenet/inference/gpu/int8/infer_detections.py
+++ b/models/object_detection/tensorflow/ssd-mobilenet/inference/gpu/int8/infer_detections.py
@@ -20,6 +20,7 @@
 from __future__ import division
 import sys
 import tensorflow as tf
+from tensorflow.core.protobuf import rewriter_config_pb2
 from tensorflow.python.data.experimental import parallel_interleave
 from tensorflow.python.data.experimental import map_and_batch
 from tensorflow.python.framework import dtypes
@@ -112,6 +113,8 @@ def __init__(self):
     arg_parser.add_argument("--benchmark",
                             help='Run in benchmark mode.',
                             dest='benchmark', action='store_true')                        
+    arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
+                            help='enable OneDNN Graph', action='store_true')
 
     # parse the arguments
     self.args = arg_parser.parse_args()
@@ -119,6 +122,8 @@ def __init__(self):
     self.config = tf.compat.v1.ConfigProto()
     # self.config.intra_op_parallelism_threads = self.args.num_intra_threads
     # self.config.inter_op_parallelism_threads = self.args.num_inter_threads
+    if self.args.onednn_graph:
+      self.config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
     self.config.use_per_session_threads = 1
 
     self.load_graph()
diff --git a/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py b/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
index 17c501077..4b84c0709 100644
--- a/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
+++ b/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
@@ -27,6 +27,7 @@
 from argparse import ArgumentParser
 from inference.coco_detection_evaluator import CocoDetectionEvaluator
 from inference.coco_label_map import category_map
+from tensorflow.core.protobuf import rewriter_config_pb2
 
 IMAGE_SIZE = 300
 COCO_NUM_VAL_IMAGES = 4952
@@ -118,6 +119,8 @@ def __init__(self):
     arg_parser.add_argument('-w', "--warmup_iter",
                             help='For accuracy measurement only.',
                             dest='warmup_iter', default=200, type=int)
+    arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
+                            help='enable OneDNN Graph', action='store_true')
 
     # parse the arguments
     self.args = arg_parser.parse_args()
@@ -125,6 +128,8 @@ def __init__(self):
     self.config = tf.compat.v1.ConfigProto()
     self.config.intra_op_parallelism_threads = self.args.num_intra_threads
     self.config.inter_op_parallelism_threads = self.args.num_inter_threads
+    if self.args.onednn_graph:
+      self.config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
     self.config.use_per_session_threads = 1
 
     self.load_graph()
diff --git a/models/object_detection/tensorflow/ssd-resnet34/inference/int8/infer_detections.py b/models/object_detection/tensorflow/ssd-resnet34/inference/int8/infer_detections.py
index 12ca0f040..9cddfce3a 100644
--- a/models/object_detection/tensorflow/ssd-resnet34/inference/int8/infer_detections.py
+++ b/models/object_detection/tensorflow/ssd-resnet34/inference/int8/infer_detections.py
@@ -43,6 +43,8 @@
 from PIL import Image
 import coco_constants
 
+from tensorflow.core.protobuf import rewriter_config_pb2
+
 class MyEncoder(json.JSONEncoder):
    def default(self, obj):
      if isinstance(obj, (numpy.int_, numpy.intc, numpy.intp, numpy.int8,
@@ -106,6 +108,9 @@ def __init__(self):
                             help="Number of steps",
                             dest='steps', type=int, default=800)
 
+    arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
+                            help='enable OneDNN Graph', action='store_true')
+
     # parse the arguments
     self.args = arg_parser.parse_args()
 
@@ -113,6 +118,9 @@ def __init__(self):
     self.config = tf.compat.v1.ConfigProto()
     self.config.intra_op_parallelism_threads = self.args.num_intra_threads
     self.config.inter_op_parallelism_threads = self.args.num_inter_threads
+    if self.args.onednn_graph:
+      self.config.graph_options.rewrite_options.constant_folding = rewriter_config_pb2.RewriterConfig.OFF
+
 
     if self.args.batch_size == -1:
       self.args.batch_size = 64
@@ -319,7 +327,7 @@ def cocoval(self, detected_json, eval_json):
 
   def run_inference_for_eval(self, graph):
     with graph.as_default():
-      with tf.compat.v1.Session() as sess:
+      with tf.compat.v1.Session(config=self.config) as sess:
 
         num_iter = self.num_batches
 
diff --git a/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py b/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
index 796f170c6..9b6c4bc90 100644
--- a/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
+++ b/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
@@ -335,6 +335,7 @@ def __init__(
             self.emb_l, _, _ = self.create_emb(m_spa, ln_emb, None, None, np_init_emb_weight)
         self.loss_fn = torch.nn.BCELoss(reduction="mean")
         self.emb_args_preprocessing = 0
+        self.load_data_time = 0
                 
     def apply_mlp(self, x, layers):
         # approach 1: use ModuleList
@@ -720,15 +721,6 @@ def distributed_forward(self, dense_x, lS_o, lS_i, is_train=False):
 
         lS_o_dense = [lS_o[i] for i in self.ln_emb_dense]
         lS_o_sparse = [lS_o[i] for i in self.ln_emb_sparse]
-        # shuffle_start = time.time()
-        # #lS_i_sparse = ext_dist.shuffle_data(lS_i_sparse)
-        # lS_i_sparse = torch.cat(lS_i_sparse)
-        # output_lS_i_sparse = lS_i_sparse.new_empty(lS_i_sparse.size())
-        # #print("cat time:", 1000*(time.time() - shuffle_start))
-        # req = ext_dist.dist.all_to_all_single(output_lS_i_sparse, lS_i_sparse)
-        # lS_i_sparse = output_lS_i_sparse.reshape(ext_dist.my_size, -1)
-
-        # g_i_sparse = [lS_i_sparse[:, i * local_batch_size:(i + 1) * local_batch_size].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
 
         global first_iteration_for_train
         if first_iteration_for_train or not is_train: 
@@ -752,13 +744,11 @@ def distributed_forward(self, dense_x, lS_o, lS_i, is_train=False):
         ly_sparse = self.apply_emb(dlrm.emb_sparse, g_o_sparse, g_i_sparse)
         a2a_req = ext_dist.alltoall(ly_sparse, self.n_sparse_emb_per_rank)
 
-        if is_train:
-            load_data(buffer_num, args.bf16)
-
         #dense embedding 
         ly_dense =  self.apply_emb(dlrm.emb_dense, lS_o_dense, lS_i_dense)
         # bottom mlp
         x = self.apply_mlp(dense_x, self.bot_l)
+        # ext_dist.barrier()
         ly_sparse = a2a_req.wait()
         _ly = []
         vector_lenght = 128
@@ -768,37 +758,12 @@ def distributed_forward(self, dense_x, lS_o, lS_i, is_train=False):
         _ly = list(ly_dense) + list(_ly)
 
         a2a_ind_req = None #ovlerlap the a2a_ind_req with interaction/top_mlp
-        if is_train:
-            #get global index for sparse embedding
-            (X_next, lS_o_next, lS_i_next, T_next, W_next, CBPP_next) = data_buffer[0]
-            lS_i_dense_next = [lS_i_next[i] for i in self.ln_emb_dense]
-            lS_i_sparse_next = [lS_i_next[i] for i in self.ln_emb_sparse]
-            if ext_dist.my_size > len(self.ln_emb_sparse):
-                num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
-                lS_i_sparse_next = torch.cat([lS_i_sparse_next for _ in range(num_split_grps) ])
-            lS_i_sparse_next = torch.cat(lS_i_sparse_next)
-            output_lS_i_sparse_next = lS_i_sparse_next.new_empty(lS_i_sparse_next.size())
-            # output_ind = self.output_ind.get_data_base_on_input(lS_i_sparse_next, lS_i_sparse_next.size())
-            a2a_ind_req = ext_dist.dist.all_to_all_single(output_lS_i_sparse_next, lS_i_sparse_next, async_op=True)
 
         # interactions
         z = self.interact_features(x, _ly)
         # top mlp
         p = self.apply_mlp(z, self.top_l)
 
-        if is_train:
-            a2a_ind_req.wait()
-            lS_i_sparse_next = output_lS_i_sparse_next.reshape(ext_dist.my_size, -1)
-            batch_size_next = X_next.size()[0]
-            # g_i_sparse = [lS_i_sparse_next[:, i * batch_size_next:(i + 1) * batch_size_next].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
-            g_i_sparse_next = [lS_i_sparse_next[:, i * batch_size_next:(i + 1) * batch_size_next].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
-            # lS_i_sparse_next = torch.cat(g_i_sparse)
-            # import pdb
-            # pdb.set_trace()
-            # for i in self.ln_emb_sparse:
-            #    lS_i_next[i] = lS_i_sparse_next[i]
-            data_buffer[0] = (X_next, lS_o_next, [lS_i_dense_next, g_i_sparse_next], T_next, W_next, CBPP_next)
-
         # clamp output if needed
         if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
             z = torch.clamp(
@@ -806,6 +771,28 @@ def distributed_forward(self, dense_x, lS_o, lS_i, is_train=False):
             )                                                                         
         else:
             z = p
+        if is_train:
+            with record_function("dist_data_loader"):
+                start = time_wrap()
+                # ext_dist.barrier()
+                load_data(buffer_num, args.bf16)
+                #get global index for sparse embedding
+                (X_next, lS_o_next, lS_i_next, T_next, W_next, CBPP_next) = data_buffer[0]
+                lS_i_dense_next = [lS_i_next[i] for i in self.ln_emb_dense]
+                lS_i_sparse_next = [lS_i_next[i] for i in self.ln_emb_sparse]
+                if ext_dist.my_size > len(self.ln_emb_sparse):
+                    num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
+                    lS_i_sparse_next = torch.cat([lS_i_sparse_next for _ in range(num_split_grps) ])
+                lS_i_sparse_next = torch.cat(lS_i_sparse_next)
+                output_lS_i_sparse_next = lS_i_sparse_next.new_empty(lS_i_sparse_next.size())
+                a2a_ind_req = ext_dist.dist.all_to_all_single(output_lS_i_sparse_next, lS_i_sparse_next, async_op=True)
+                a2a_ind_req.wait()
+                lS_i_sparse_next = output_lS_i_sparse_next.reshape(ext_dist.my_size, -1)
+                batch_size_next = X_next.size()[0]
+                g_i_sparse_next = [lS_i_sparse_next[:, i * batch_size_next:(i + 1) * batch_size_next].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
+                data_buffer[0] = (X_next, lS_o_next, [lS_i_dense_next, g_i_sparse_next], T_next, W_next, CBPP_next)
+                # ext_dist.barrier()
+                self.load_data_time += time_wrap() - start
         return z
  
 
@@ -1592,7 +1579,8 @@ def trace_handler(prof):
 
                         lr_scheduler.step()
 
-                        t2 = time_wrap() - dlrm.emb_args_preprocessing
+                        t2 = time_wrap() - dlrm.emb_args_preprocessing - dlrm.load_data_time
+                        dlrm.load_data_time = 0
                         dlrm.emb_args_preprocessing = 0 
                         total_train_time_wo_dl_eval += (t2 - t1)
                         total_time += t2 - t1
@@ -1656,6 +1644,7 @@ def trace_handler(prof):
                                 best_acc_test = model_metrics_dict["test_acc"]
                             eval_end = time.time()
                             dlrm.emb_args_preprocessing = 0
+                            dlrm.load_data_time = 0
                             if ext_dist.my_size > 1 and ext_dist.dist.get_rank() ==1 :
                                 print("Evauation at {} iteration using {} s".format(j+1, eval_end- eval_begin))
                             if (
diff --git a/models/recommendation/pytorch/dlrm/product/extend_distributed.py b/models/recommendation/pytorch/dlrm/product/extend_distributed.py
index 66e13ad87..feebf15af 100644
--- a/models/recommendation/pytorch/dlrm/product/extend_distributed.py
+++ b/models/recommendation/pytorch/dlrm/product/extend_distributed.py
@@ -292,6 +292,7 @@ def forward(ctx, a2ai, *inputs):
 
     @staticmethod
     def backward(ctx, *grad_output):
+        # dist.barrier()
         global myreq
         #print("All2All_Req:backward")
         a2ai = ctx.a2ai
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/LICENSE b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/LICENSE
new file mode 100644
index 000000000..f6a14ae7a
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/LICENSE
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2023] [Intel Corporation]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/README.md b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/README.md
new file mode 100644
index 000000000..37f3d1096
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/README.md
@@ -0,0 +1,26 @@
+# MEMREC Encoder: 
+
+MEMREC Encoder encodes categorical tokens into a large binary vector using wyhash.
+
+## Installation
+
+To compile and install the module use:
+
+`python setup.py install --user`
+
+## Instructions
+
+Usage:
+
+`import memrec_encoder`
+
+`memrec_encoder.wyh_hash_array_sparse(input_list, D, n_bit_hash, featureCnt, K, kw, hash_tech_id)`
+
+The arguments are as follows:
+- `input_list` - A numpy array containing the data to be hashed. 
+- `n_bit_hash` - For hashing function to generate n bit hases.
+- `D` - The encoding dimension to be used. Currently same for both embedding tables.
+- `featureCnt` - Feature count, i.e. number of columns in the data (26 for Criteo TB).
+- `K and kw` - Number of hashes for each embedding table.
+- `hash_tech_id` - 2 for wy hash.
+
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/THIRD-PARTY-LICENSES b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/THIRD-PARTY-LICENSES
new file mode 100644
index 000000000..940941214
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/THIRD-PARTY-LICENSES
@@ -0,0 +1,68 @@
+MEMREC Third Party Programs File
+
+This file contains the list of third party software ("third party programs")
+contained in the Intel software and their required notices and/or license
+terms. This third party software, even if included with the distribution of the
+Intel software, may be governed by separate license terms, including without
+limitation, third party license terms, other Intel software license terms, and
+open source software license terms. These separate license terms govern your use
+of the third party programs as set forth in the "third-party-programs.txt" or
+other similarly named text file.
+ 
+Third party programs and their corresponding required notices and/or license
+terms are listed below.
+
+--------------------------------------------------------------------------------
+
+1.  wyhash
+
+	This is free and unencumbered software released into the public domain.
+
+	Anyone is free to copy, modify, publish, use, compile, sell, or
+	distribute this software, either in source code form or as a compiled
+	binary, for any purpose, commercial or non-commercial, and by any
+	means.
+
+	In jurisdictions that recognize copyright laws, the author or authors
+	of this software dedicate any and all copyright interest in the
+	software to the public domain. We make this dedication for the benefit
+	of the public at large and to the detriment of our heirs and
+	successors. We intend this dedication to be an overt act of
+	relinquishment in perpetuity of all present and future rights to this
+	software under copyright law.
+
+	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+	MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+	IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+	OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+	ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+	OTHER DEALINGS IN THE SOFTWARE.
+
+	For more information, please refer to <http://unlicense.org/>
+
+2. DLRM
+
+	MIT License
+
+	Copyright (c) Facebook, Inc. and its affiliates.
+
+	Permission is hereby granted, free of charge, to any person obtaining a copy
+	of this software and associated documentation files (the "Software"), to deal
+	in the Software without restriction, including without limitation the rights
+	to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+	copies of the Software, and to permit persons to whom the Software is
+	furnished to do so, subject to the following conditions:
+
+	The above copyright notice and this permission notice shall be included in all
+	copies or substantial portions of the Software.
+
+	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+	IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+	FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+	AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+	LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+	OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+	SOFTWARE.
+	
+--------------------------------------------------------------------------------
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/.gitignore b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/.gitignore
new file mode 100644
index 000000000..9acb272f7
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/.gitignore
@@ -0,0 +1,5 @@
+build
+dist
+memrec_encoder.egg-info
+mmh3.cpython-38-x86_64-linux-gnu.so
+mmh3.egg-info
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/memrec_encoder.cpp b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/memrec_encoder.cpp
new file mode 100644
index 000000000..9ccd25ab9
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/memrec_encoder.cpp
@@ -0,0 +1,256 @@
+// Copyright (c) 2023 Intel Corporation
+
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+
+//    http://www.apache.org/licenses/LICENSE-2.0
+
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+#ifndef PY_SSIZE_T_CLEAN
+#define PY_SSIZE_T_CLEAN
+#endif
+#include <stdlib.h> 
+#include <map>
+#include <ctime>
+#include <Python.h>
+#include <stdio.h>
+#include <iostream>
+#include <string.h>
+#include <math.h>
+#include <numpy/arrayobject.h>
+#include  "wyhash.h"
+
+#if defined(_MSC_VER)
+typedef signed __int8 int8_t;
+typedef signed __int32 int32_t;
+typedef signed __int64 int64_t;
+typedef unsigned __int8 uint8_t;
+typedef unsigned __int32 uint32_t;
+typedef unsigned __int64 uint64_t;
+// Other compilers
+#else    // defined(_MSC_VER)
+#include <stdint.h>
+#endif // !defined(_MSC_VER)
+uint32_t seeds[20] = {7186053, 6387688, 5937987, 2014911, 8420444, 7735429, 7189516, 6746271, 6122425, 3363721, 1344626, 3883505, 357814, 7959252, 5356618, 6020406, 3523941, 3551544, 8555633, 9130236};
+#define SETITEM(ii, jj) (*(npy_int8*)((PyArray_DATA(output_array) +               \
+                                      (ii) * PyArray_STRIDES(output_array)[0] +   \
+                                      (jj) * PyArray_STRIDES(output_array)[1])))
+
+#define SETITEM_(ii, jj) (*(npy_int32*)((PyArray_DATA(output_array) +               \
+                                      (ii) * PyArray_STRIDES(output_array)[0] +   \
+                                      (jj) * PyArray_STRIDES(output_array)[1])))
+
+#define SETITEM_TE(ii, jj) (*(npy_int32*)((PyArray_DATA(output_array_te) +               \
+                                      (ii) * PyArray_STRIDES(output_array_te)[0] +   \
+                                      (jj) * PyArray_STRIDES(output_array_te)[1])))
+
+#define SETITEM_WE(ii, jj) (*(npy_int32*)((PyArray_DATA(output_array_we) +               \
+                                      (ii) * PyArray_STRIDES(output_array_we)[0] +   \
+                                      (jj) * PyArray_STRIDES(output_array_we)[1])))
+
+#define SETITEM__(ii, jj) (*(npy_float32*)((PyArray_DATA(output_array) +               \
+                                      (ii) * PyArray_STRIDES(output_array)[0] +   \
+                                      (jj) * PyArray_STRIDES(output_array)[1])))
+
+static PyObject *
+wyh_hash_array_sparse(PyObject *self, PyObject *args)
+{
+
+    int hash_tech_id;
+    PyArrayObject *input_list;
+    PyArrayObject *output_array;
+    
+    int target_str_len, d, num_features, k, kw, num_ones_te, num_ones_we;
+
+    if (!PyArg_ParseTuple
+    (args, "O!iiiiii", 
+     &PyArray_Type, &input_list,
+     &d, &target_str_len, &num_features, &k, &kw, &hash_tech_id))
+    {
+        return NULL;
+    }
+
+    Py_ssize_t nrows, ncols, nseeds_te, nseeds_we;
+    nrows = PyArray_SHAPE(input_list)[0];
+    ncols = PyArray_SHAPE(input_list)[1];
+    num_ones_te = num_features*k;
+    num_ones_we = num_features*kw;
+    
+    npy_intp dims_[1] = { nrows*(num_ones_te+num_ones_we)};
+    output_array = (PyArrayObject*) PyArray_ZEROS(1, dims_, NPY_INT, 0);
+
+    uint32_t result[1];
+    uint64_t _wyp[4];
+    int new_jj_te=0;
+    int new_jj_we=0;
+
+    uint32_t seed = seeds[0];				
+    Py_ssize_t ix[2];
+    //make_secret(seed,_wyp);
+    std::clock_t time_4 = std::clock();
+
+    for (int jj=0; jj<ncols; jj++)
+    {
+        bool use_hot_hashes = true;
+        int top_n_hashes = 10; //power of 2
+        uint64_t hot_hashes[top_n_hashes];
+        uint64_t h;
+
+        for (int i=0; i<top_n_hashes; i++)
+        {            
+            uint16_t token_int=jj*430000000 + i; 
+            hot_hashes[i]=wyhash(&token_int, target_str_len, seed, _wyp);
+        }
+        for (int ii=0; ii<nrows; ii++)
+        {            
+            uint16_t *item = (uint16_t *) PyArray_GETPTR2(input_list, ii, jj);
+            if ((*item < top_n_hashes) && use_hot_hashes)
+            {
+              h = hot_hashes[*item];
+            }
+            else
+            {
+              uint16_t token_int = jj*430000000 + *item; 
+              h=wyhash(&token_int, target_str_len, seed, _wyp);
+            }
+
+            
+            uint32_t msw, lsw;
+            msw = (h & 0xFFFFFFFF00000000) >> 32;
+            lsw = h & 0x00000000FFFFFFFF;
+            ix[0] = (msw) % d;
+            ix[1] = (lsw) % d; //d vs its value 
+
+            uint32_t *item_address_0 = (uint32_t *) PyArray_GETPTR1(output_array, ii*k*ncols+jj);
+				    uint32_t *item_address_1 = (uint32_t *) PyArray_GETPTR1(output_array, nrows*num_ones_te+ii*kw*ncols+jj);
+            *item_address_0 = ix[0];
+				    *item_address_1 = ix[1];
+
+            int num_more_hashes = k+kw-2;
+            uint64_t more_hashes[num_more_hashes];
+            
+            if (num_more_hashes)
+            {
+              //create more hashes
+              int more_hashes_idx = 0;
+              for (int m=0; m<ceil((float)num_more_hashes/2); m++)
+              {
+                uint16_t token_int = jj*430000000 + *item; 
+                h=wyhash(&token_int, target_str_len, seeds[m+1], _wyp);
+                msw = (h & 0xFFFFFFFF00000000) >> 32;
+                more_hashes[more_hashes_idx] = (msw) % d;
+                more_hashes_idx++;
+                lsw = h & 0x00000000FFFFFFFF;
+                if (more_hashes_idx < num_more_hashes) more_hashes[more_hashes_idx] = (lsw) % d;
+                more_hashes_idx++;
+              }
+              //store generated hashes at appropriate places in the output array
+              int store_idx = 0;
+              for (int this_k=1; this_k<k; this_k++) //this_k = 0 has already been processed
+              {
+                uint32_t *store_here = (uint32_t *) PyArray_GETPTR1(output_array, ii*k*ncols + this_k*ncols + jj);
+                *store_here = more_hashes[store_idx];
+                store_idx++;
+              }
+              for (int this_kw=1; this_kw<kw; this_kw++) //this_k = 0 has already been processed
+              {
+                uint32_t *store_here = (uint32_t *) PyArray_GETPTR1(output_array, nrows*num_ones_te + ii*kw*ncols + this_kw*ncols + jj);
+                *store_here = more_hashes[store_idx];
+                store_idx++;
+              }
+            }
+            
+        }
+    }
+
+    return PyArray_Return(output_array);
+}
+
+
+struct module_state {
+  PyObject *error;
+};
+    
+#if PY_MAJOR_VERSION >= 3
+#define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))
+#else
+#define GETSTATE(m) (&_state)
+static struct module_state _state;
+#endif
+
+static PyMethodDef MEMethods[] = {
+    {"wyh_hash_array_sparse", (PyCFunction)wyh_hash_array_sparse, METH_VARARGS},
+    {NULL, NULL, 0, NULL}
+};
+
+#if PY_MAJOR_VERSION >= 3
+
+static int me_traverse(PyObject *m, visitproc visit, void *arg) {
+    Py_VISIT(GETSTATE(m)->error);
+    return 0;
+}
+
+static int me_clear(PyObject *m) {
+    Py_CLEAR(GETSTATE(m)->error);
+    return 0;
+}
+
+static struct PyModuleDef memrec_encodermodule = {
+    PyModuleDef_HEAD_INIT,
+    "memrec_encoder",
+    "memrec_encoder is the implementation of memrec embedding encoder using WY hash",
+    sizeof(struct module_state),
+    MEMethods,
+    NULL,
+    me_traverse,
+    me_clear,
+    NULL
+};
+
+#define INITERROR return NULL
+
+extern "C" {
+PyMODINIT_FUNC
+PyInit_memrec_encoder(void)
+
+#else // PY_MAJOR_VERSION >= 3
+#define INITERROR return
+
+extern "C" {
+void
+initmemrec_encoder(void)
+#endif // PY_MAJOR_VERSION >= 3
+
+{
+#if PY_MAJOR_VERSION >= 3
+    PyObject *module = PyModule_Create(&memrec_encodermodule);
+    import_array();
+#else
+    PyObject *module = Py_InitModule("memrec_encoder", MEMethods);
+#endif
+
+    if (module == NULL)
+        INITERROR;
+
+    PyModule_AddStringConstant(module, "__version__", "3.0.0");
+
+    struct module_state *st = GETSTATE(module);
+
+    st->error = PyErr_NewException((char *) "memrec_encoder.Error", NULL, NULL);
+    if (st->error == NULL) {
+        Py_DECREF(module);
+        INITERROR;
+    }
+
+#if PY_MAJOR_VERSION >= 3
+    return module;
+#endif
+}
+} // extern "C"
\ No newline at end of file
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/setup.py b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/setup.py
new file mode 100644
index 000000000..e269aabf8
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/setup.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import platform
+import sys
+
+import numpy as np
+from setuptools import Extension, setup
+
+COMPILE_OPTIONS = []
+LINK_OPTIONS = []
+
+
+def is_new_osx():
+    """Check whether we're on OSX >= 10.7"""
+    if sys.platform != "darwin":
+        return False
+    mac_ver = platform.mac_ver()[0]
+    if mac_ver.startswith("10"):
+        minor_version = int(mac_ver.split(".")[1])
+        return minor_version >= 7
+    return False
+
+
+if is_new_osx():
+    # On Mac, use libc++ because Apple deprecated use of
+    # libstdc
+    COMPILE_OPTIONS.append("-stdlib=libc++")
+    LINK_OPTIONS.append("-lc++")
+    # g++ (used by unix compiler on mac) links to libstdc++ as a default lib.
+    # See: https://stackoverflow.com/questions/1653047/avoid-linking-to-libstdc
+    LINK_OPTIONS.append("-nodefaultlibs")
+
+memrec_encodermodule = Extension(
+    "memrec_encoder",
+    sources=["memrec_encoder.cpp"],
+    extra_compile_args=COMPILE_OPTIONS,
+    extra_link_args=LINK_OPTIONS,
+    include_dirs=[np.get_include()]
+)
+
+setup(
+    name="memrec_encoder",
+    version="3.0.0",
+    description="Memrec embedding encoder module",
+    ext_modules=[memrec_encodermodule],
+    keywords="Memrec Embedding Encoder",
+    long_description_content_type="text/markdown",
+)
diff --git a/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/wyhash.h b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/wyhash.h
new file mode 100644
index 000000000..7df33d4d3
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/wyhash.h
@@ -0,0 +1,231 @@
+// This is free and unencumbered software released into the public domain under The Unlicense (http://unlicense.org/)
+// main repo: https://github.com/wangyi-fudan/wyhash
+// author: 王一 Wang Yi <godspeed_china@yeah.net>
+// contributors: Reini Urban, Dietrich Epp, Joshua Haberman, Tommy Ettinger, Daniel Lemire, Otmar Ertl, cocowalla, leo-yuriev, Diego Barrios Romero, paulie-g, dumblob, Yann Collet, ivte-ms, hyb, James Z.M. Gao, easyaspi314 (Devin), TheOneric
+
+/* quick example:
+   string s="fjsakfdsjkf";
+   uint64_t hash=wyhash(s.c_str(), s.size(), 0, _wyp);
+*/
+
+#ifndef wyhash_final_version_4
+#define wyhash_final_version_4
+
+#ifndef WYHASH_CONDOM
+//protections that produce different results:
+//1: normal valid behavior
+//2: extra protection against entropy loss (probability=2^-63), aka. "blind multiplication"
+#define WYHASH_CONDOM 1
+#endif
+
+#ifndef WYHASH_32BIT_MUM
+//0: normal version, slow on 32 bit systems
+//1: faster on 32 bit systems but produces different results, incompatible with wy2u0k function
+#define WYHASH_32BIT_MUM 0  
+#endif
+
+//includes
+#include <stdint.h>
+#include <string.h>
+#if defined(_MSC_VER) && defined(_M_X64)
+  #include <intrin.h>
+  #pragma intrinsic(_umul128)
+#endif
+
+//likely and unlikely macros
+#if defined(__GNUC__) || defined(__INTEL_COMPILER) || defined(__clang__)
+  #define _likely_(x)  __builtin_expect(x,1)
+  #define _unlikely_(x)  __builtin_expect(x,0)
+#else
+  #define _likely_(x) (x)
+  #define _unlikely_(x) (x)
+#endif
+
+//128bit multiply function
+static inline uint64_t _wyrot(uint64_t x) { return (x>>32)|(x<<32); }
+static inline void _wymum(uint64_t *A, uint64_t *B){
+#if(WYHASH_32BIT_MUM)
+  uint64_t hh=(*A>>32)*(*B>>32), hl=(*A>>32)*(uint32_t)*B, lh=(uint32_t)*A*(*B>>32), ll=(uint64_t)(uint32_t)*A*(uint32_t)*B;
+  #if(WYHASH_CONDOM>1)
+  *A^=_wyrot(hl)^hh; *B^=_wyrot(lh)^ll;
+  #else
+  *A=_wyrot(hl)^hh; *B=_wyrot(lh)^ll;
+  #endif
+#elif defined(__SIZEOF_INT128__)
+  __uint128_t r=*A; r*=*B; 
+  #if(WYHASH_CONDOM>1)
+  *A^=(uint64_t)r; *B^=(uint64_t)(r>>64);
+  #else
+  *A=(uint64_t)r; *B=(uint64_t)(r>>64);
+  #endif
+#elif defined(_MSC_VER) && defined(_M_X64)
+  #if(WYHASH_CONDOM>1)
+  uint64_t  a,  b;
+  a=_umul128(*A,*B,&b);
+  *A^=a;  *B^=b;
+  #else
+  *A=_umul128(*A,*B,B);
+  #endif
+#else
+  uint64_t ha=*A>>32, hb=*B>>32, la=(uint32_t)*A, lb=(uint32_t)*B, hi, lo;
+  uint64_t rh=ha*hb, rm0=ha*lb, rm1=hb*la, rl=la*lb, t=rl+(rm0<<32), c=t<rl;
+  lo=t+(rm1<<32); c+=lo<t; hi=rh+(rm0>>32)+(rm1>>32)+c;
+  #if(WYHASH_CONDOM>1)
+  *A^=lo;  *B^=hi;
+  #else
+  *A=lo;  *B=hi;
+  #endif
+#endif
+}
+
+//multiply and xor mix function, aka MUM
+static inline uint64_t _wymix(uint64_t A, uint64_t B){ _wymum(&A,&B); return A^B; }
+
+//endian macros
+#ifndef WYHASH_LITTLE_ENDIAN
+  #if defined(_WIN32) || defined(__LITTLE_ENDIAN__) || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
+    #define WYHASH_LITTLE_ENDIAN 1
+  #elif defined(__BIG_ENDIAN__) || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+    #define WYHASH_LITTLE_ENDIAN 0
+  #else
+    #warning could not determine endianness! Falling back to little endian.
+    #define WYHASH_LITTLE_ENDIAN 1
+  #endif
+#endif
+
+//read functions
+#if (WYHASH_LITTLE_ENDIAN)
+static inline uint64_t _wyr8(const uint8_t *p) { uint64_t v; memcpy(&v, p, 8); return v;}
+static inline uint64_t _wyr4(const uint8_t *p) { uint32_t v; memcpy(&v, p, 4); return v;}
+#elif defined(__GNUC__) || defined(__INTEL_COMPILER) || defined(__clang__)
+static inline uint64_t _wyr8(const uint8_t *p) { uint64_t v; memcpy(&v, p, 8); return __builtin_bswap64(v);}
+static inline uint64_t _wyr4(const uint8_t *p) { uint32_t v; memcpy(&v, p, 4); return __builtin_bswap32(v);}
+#elif defined(_MSC_VER)
+static inline uint64_t _wyr8(const uint8_t *p) { uint64_t v; memcpy(&v, p, 8); return _byteswap_uint64(v);}
+static inline uint64_t _wyr4(const uint8_t *p) { uint32_t v; memcpy(&v, p, 4); return _byteswap_ulong(v);}
+#else
+static inline uint64_t _wyr8(const uint8_t *p) {
+  uint64_t v; memcpy(&v, p, 8);
+  return (((v >> 56) & 0xff)| ((v >> 40) & 0xff00)| ((v >> 24) & 0xff0000)| ((v >>  8) & 0xff000000)| ((v <<  8) & 0xff00000000)| ((v << 24) & 0xff0000000000)| ((v << 40) & 0xff000000000000)| ((v << 56) & 0xff00000000000000));
+}
+static inline uint64_t _wyr4(const uint8_t *p) {
+  uint32_t v; memcpy(&v, p, 4);
+  return (((v >> 24) & 0xff)| ((v >>  8) & 0xff00)| ((v <<  8) & 0xff0000)| ((v << 24) & 0xff000000));
+}
+#endif
+static inline uint64_t _wyr3(const uint8_t *p, size_t k) { return (((uint64_t)p[0])<<16)|(((uint64_t)p[k>>1])<<8)|p[k-1];}
+//wyhash main function
+static inline uint64_t wyhash(const void *key, size_t len, uint64_t seed, const uint64_t *secret){
+  const uint8_t *p=(const uint8_t *)key; seed^=_wymix(seed^secret[0],secret[1]);	uint64_t	a,	b;
+  if(_likely_(len<=16)){
+    if(_likely_(len>=4)){ a=(_wyr4(p)<<32)|_wyr4(p+((len>>3)<<2)); b=(_wyr4(p+len-4)<<32)|_wyr4(p+len-4-((len>>3)<<2)); }
+    else if(_likely_(len>0)){ a=_wyr3(p,len); b=0;}
+    else a=b=0;
+  }
+  else{
+    size_t i=len; 
+    if(_unlikely_(i>48)){
+      uint64_t see1=seed, see2=seed;
+      do{
+        seed=_wymix(_wyr8(p)^secret[1],_wyr8(p+8)^seed);
+        see1=_wymix(_wyr8(p+16)^secret[2],_wyr8(p+24)^see1);
+        see2=_wymix(_wyr8(p+32)^secret[3],_wyr8(p+40)^see2);
+        p+=48; i-=48;
+      }while(_likely_(i>48));
+      seed^=see1^see2;
+    }
+    while(_unlikely_(i>16)){  seed=_wymix(_wyr8(p)^secret[1],_wyr8(p+8)^seed);  i-=16; p+=16;  }
+    a=_wyr8(p+i-16);  b=_wyr8(p+i-8);
+  }
+  a^=secret[1]; b^=seed;  _wymum(&a,&b);
+  return  _wymix(a^secret[0]^len,b^secret[1]);
+}
+
+//the default secret parameters
+static const uint64_t _wyp[4] = {0xa0761d6478bd642full, 0xe7037ed1a0b428dbull, 0x8ebc6af09c88c6e3ull, 0x589965cc75374cc3ull};
+
+//a useful 64bit-64bit mix function to produce deterministic pseudo random numbers that can pass BigCrush and PractRand
+static inline uint64_t wyhash64(uint64_t A, uint64_t B){ A^=0xa0761d6478bd642full; B^=0xe7037ed1a0b428dbull; _wymum(&A,&B); return _wymix(A^0xa0761d6478bd642full,B^0xe7037ed1a0b428dbull);}
+
+//The wyrand PRNG that pass BigCrush and PractRand
+static inline uint64_t wyrand(uint64_t *seed){ *seed+=0xa0761d6478bd642full; return _wymix(*seed,*seed^0xe7037ed1a0b428dbull);}
+
+//convert any 64 bit pseudo random numbers to uniform distribution [0,1). It can be combined with wyrand, wyhash64 or wyhash.
+static inline double wy2u01(uint64_t r){ const double _wynorm=1.0/(1ull<<52); return (r>>12)*_wynorm;}
+
+//convert any 64 bit pseudo random numbers to APPROXIMATE Gaussian distribution. It can be combined with wyrand, wyhash64 or wyhash.
+static inline double wy2gau(uint64_t r){ const double _wynorm=1.0/(1ull<<20); return ((r&0x1fffff)+((r>>21)&0x1fffff)+((r>>42)&0x1fffff))*_wynorm-3.0;}
+
+#ifdef	WYTRNG
+#include <sys/time.h>
+//The wytrand true random number generator, passed BigCrush.
+static inline uint64_t wytrand(uint64_t *seed){
+	struct	timeval	t;	gettimeofday(&t,0);
+	uint64_t	teed=(((uint64_t)t.tv_sec)<<32)|t.tv_usec;
+	teed=_wymix(teed^_wyp[0],*seed^_wyp[1]);
+	*seed=_wymix(teed^_wyp[0],_wyp[2]);
+	return _wymix(*seed,*seed^_wyp[3]);
+}
+#endif
+
+#if(!WYHASH_32BIT_MUM)
+//fast range integer random number generation on [0,k) credit to Daniel Lemire. May not work when WYHASH_32BIT_MUM=1. It can be combined with wyrand, wyhash64 or wyhash.
+static inline uint64_t wy2u0k(uint64_t r, uint64_t k){ _wymum(&r,&k); return k; }
+#endif
+
+//make your own secret
+static inline void make_secret(uint64_t seed, uint64_t *secret){
+  uint8_t c[] = {15, 23, 27, 29, 30, 39, 43, 45, 46, 51, 53, 54, 57, 58, 60, 71, 75, 77, 78, 83, 85, 86, 89, 90, 92, 99, 101, 102, 105, 106, 108, 113, 114, 116, 120, 135, 139, 141, 142, 147, 149, 150, 153, 154, 156, 163, 165, 166, 169, 170, 172, 177, 178, 180, 184, 195, 197, 198, 201, 202, 204, 209, 210, 212, 216, 225, 226, 228, 232, 240 };
+  for(size_t i=0;i<4;i++){
+    uint8_t ok;
+    do{
+      ok=1; secret[i]=0;
+      for(size_t j=0;j<64;j+=8) secret[i]|=((uint64_t)c[wyrand(&seed)%sizeof(c)])<<j;
+      if(secret[i]%2==0){ ok=0; continue; }
+      for(size_t j=0;j<i;j++) {
+#if defined(__GNUC__) || defined(__INTEL_COMPILER) || defined(__clang__)
+        if(__builtin_popcountll(secret[j]^secret[i])!=32){ ok=0; break; }
+#elif defined(_MSC_VER) && defined(_M_X64)
+        if(_mm_popcnt_u64(secret[j]^secret[i])!=32){ ok=0; break; }
+#else
+        //manual popcount
+        uint64_t x = secret[j]^secret[i];
+        x -= (x >> 1) & 0x5555555555555555;
+        x = (x & 0x3333333333333333) + ((x >> 2) & 0x3333333333333333);
+        x = (x + (x >> 4)) & 0x0f0f0f0f0f0f0f0f;
+        x = (x * 0x0101010101010101) >> 56;
+        if(x!=32){ ok=0; break; }
+#endif
+      }
+    }while(!ok);
+  }
+}
+
+#endif
+
+/* The Unlicense
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org/>
+*/
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/.gitignore b/models/recommendation/pytorch/memrec_dlrm/product/.gitignore
new file mode 100644
index 000000000..bee8a64b7
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/.gitignore
@@ -0,0 +1 @@
+__pycache__
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/LICENSE b/models/recommendation/pytorch/memrec_dlrm/product/LICENSE
new file mode 100644
index 000000000..f6a14ae7a
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/LICENSE
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2023] [Intel Corporation]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/README.md b/models/recommendation/pytorch/memrec_dlrm/product/README.md
new file mode 100644
index 000000000..7f76d36df
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/README.md
@@ -0,0 +1,28 @@
+<!--- 0. Title -->
+# MEMREC: Memory Efficient Recommendation System using Alternative Representation
+
+<!-- 10. Description -->
+## Description
+
+Deep learning-based recommendation systems (e.g., DLRMs) are widely used AI models to provide high-quality personalized recommendations. Training data used for modern recommendation systems commonly includes categorical features taking on tens-of-millions of possible distinct values. These categorical tokens are typically assigned learned vector representations, that are stored in large embedding tables, on the order of 100s of GB. Storing and accessing these tables represent a substantial burden in commercial deployments. Our work proposes MEM-REC, a novel alternative representation approach for embedding tables. MEM-REC leverages bloom filters and hashing methods to encode categorical features using two cache-friendly embedding tables. The first table (token embedding) contains raw embeddings (i.e. learned vector representation), and the second table (weight embedding), which is much smaller, contains weights to scale these raw embeddings to provide better discriminative capability to each data point. We provide a detailed architecture, design and analysis of MEM-REC addressing trade-offs in accuracy and computation requirements, in comparison with state-of-the-art techniques. We show that MEM-REC can not only maintain the recommendation quality and significantly reduce the memory footprint for commercial scale recommendation models but can also improve the embedding latency. In particular, based on our results, MEM-REC compresses the MLPerf CriteoTB benchmark DLRM model size by 2900x and performs up to 3.4x faster embeddings while achieving the same AUC as that of the full uncompressed model.
+
+## Details 
+Please refer to [MEMREC reseasrch paper (to appear in ACML'23)](https://arxiv.org/pdf/2305.07205.pdf) available on arxiv.
+
+## Important MEMREC Parameters
+  - ```D```: Size of the sparse binary vector - Tier-1 embedding table
+  - ```K```: Number of Hash Functions - Tier-1 embedding table
+  - ```dw```: Size of the sparse binary vector - Tier-2 embedding table
+  - ```kw```: Number of Hash Functions - Tier-2 embedding table
+  - ```arch-mlp-bot-sparse```: MLP that learns on the top of HD-Encoded data
+  - ```arch-mlp-bot```: DLRM Dense processing MLP
+  - ```arch-mlp-top```: DLRM predictor MLP
+
+
+<!--- 80. License -->
+## License
+
+[LICENSE](../product/LICENSE)
+
+[THIRD PARTY LICENSES](../product/THIRD-PARTY-LICENSES)
+
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/THIRD-PARTY-LICENSES b/models/recommendation/pytorch/memrec_dlrm/product/THIRD-PARTY-LICENSES
new file mode 100644
index 000000000..940941214
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/THIRD-PARTY-LICENSES
@@ -0,0 +1,68 @@
+MEMREC Third Party Programs File
+
+This file contains the list of third party software ("third party programs")
+contained in the Intel software and their required notices and/or license
+terms. This third party software, even if included with the distribution of the
+Intel software, may be governed by separate license terms, including without
+limitation, third party license terms, other Intel software license terms, and
+open source software license terms. These separate license terms govern your use
+of the third party programs as set forth in the "third-party-programs.txt" or
+other similarly named text file.
+ 
+Third party programs and their corresponding required notices and/or license
+terms are listed below.
+
+--------------------------------------------------------------------------------
+
+1.  wyhash
+
+	This is free and unencumbered software released into the public domain.
+
+	Anyone is free to copy, modify, publish, use, compile, sell, or
+	distribute this software, either in source code form or as a compiled
+	binary, for any purpose, commercial or non-commercial, and by any
+	means.
+
+	In jurisdictions that recognize copyright laws, the author or authors
+	of this software dedicate any and all copyright interest in the
+	software to the public domain. We make this dedication for the benefit
+	of the public at large and to the detriment of our heirs and
+	successors. We intend this dedication to be an overt act of
+	relinquishment in perpetuity of all present and future rights to this
+	software under copyright law.
+
+	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+	MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+	IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+	OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+	ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+	OTHER DEALINGS IN THE SOFTWARE.
+
+	For more information, please refer to <http://unlicense.org/>
+
+2. DLRM
+
+	MIT License
+
+	Copyright (c) Facebook, Inc. and its affiliates.
+
+	Permission is hereby granted, free of charge, to any person obtaining a copy
+	of this software and associated documentation files (the "Software"), to deal
+	in the Software without restriction, including without limitation the rights
+	to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+	copies of the Software, and to permit persons to whom the Software is
+	furnished to do so, subject to the following conditions:
+
+	The above copyright notice and this permission notice shall be included in all
+	copies or substantial portions of the Software.
+
+	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+	IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+	FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+	AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+	LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+	OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+	SOFTWARE.
+	
+--------------------------------------------------------------------------------
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/data_loader_terabyte.py b/models/recommendation/pytorch/memrec_dlrm/product/data_loader_terabyte.py
new file mode 100644
index 000000000..298f14830
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/data_loader_terabyte.py
@@ -0,0 +1,390 @@
+# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import numpy as np
+from torch.utils.data import Dataset
+import torch
+import time
+import math
+from tqdm import tqdm
+import argparse
+import extend_distributed as ext_dist
+
+
+class DataLoader:
+    """
+    DataLoader dedicated for the Criteo Terabyte Click Logs dataset
+    """
+
+    def __init__(
+            self,
+            data_filename,
+            data_directory,
+            days,
+            batch_size,
+            max_ind_range=-1,
+            split="train",
+            drop_last_batch=False
+    ):
+        self.data_filename = data_filename
+        self.data_directory = data_directory
+        self.days = days
+        self.batch_size = batch_size
+        self.max_ind_range = max_ind_range
+
+        total_file = os.path.join(
+            data_directory,
+            data_filename + "_day_count.npz"
+        )
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"][np.array(days)]
+
+        self.length = sum(total_per_file)
+        if split == "test" or split == "val":
+            self.length = int(np.ceil(self.length / 2.))
+        self.split = split
+        self.drop_last_batch = drop_last_batch
+
+    def __iter__(self):
+        return iter(
+            _batch_generator(
+                self.data_filename, self.data_directory, self.days,
+                self.batch_size, self.split, self.drop_last_batch, self.max_ind_range
+            )
+        )
+
+    def __len__(self):
+        if self.drop_last_batch:
+            return self.length // self.batch_size
+        else:
+            return math.ceil(self.length / self.batch_size)
+
+
+def _transform_features(
+        x_int_batch, x_cat_batch, y_batch, max_ind_range, flag_input_torch_tensor=False
+):
+    if max_ind_range > 0:
+        x_cat_batch = x_cat_batch % max_ind_range
+
+    if flag_input_torch_tensor:
+        x_int_batch = torch.log(x_int_batch.clone().detach().type(torch.float) + 1)
+        x_cat_batch = x_cat_batch.clone().detach().type(torch.long)
+        y_batch = y_batch.clone().detach().type(torch.float32).view(-1, 1)
+    else:
+        x_int_batch = torch.log(torch.tensor(x_int_batch, dtype=torch.float) + 1)
+        x_cat_batch = torch.tensor(x_cat_batch, dtype=torch.long)
+        y_batch = torch.tensor(y_batch, dtype=torch.float32).view(-1, 1)
+
+    batch_size = x_cat_batch.shape[0]
+    feature_count = x_cat_batch.shape[1]
+    lS_o = torch.arange(batch_size).reshape(1, -1).repeat(feature_count, 1)
+    return x_int_batch, lS_o, x_cat_batch.t(), y_batch.view(-1, 1)
+
+
+def _batch_generator(
+        data_filename, data_directory, days, batch_size, split, drop_last, max_ind_range
+):
+    previous_file = None
+    for day in days:
+        filepath = os.path.join(
+            data_directory,
+            data_filename + "_{}_reordered.npz".format(day)
+        )
+
+        # print('Loading file: ', filepath)
+        with np.load(filepath) as data:
+            x_int = data["X_int"]
+            x_cat = data["X_cat"]
+            y = data["y"]
+
+        samples_in_file = y.shape[0]
+        batch_start_idx = 0
+        if split == "test" or split == "val":
+            length = int(np.ceil(samples_in_file / 2.))
+            if split == "test":
+                samples_in_file = length
+            elif split == "val":
+                batch_start_idx = samples_in_file - length
+
+        while batch_start_idx < samples_in_file - batch_size:
+
+            missing_samples = batch_size
+            if previous_file is not None:
+                missing_samples -= previous_file['y'].shape[0]
+
+            current_slice = slice(batch_start_idx, batch_start_idx + missing_samples)
+
+            x_int_batch = x_int[current_slice]
+            x_cat_batch = x_cat[current_slice]
+            y_batch = y[current_slice]
+
+            if previous_file is not None:
+                x_int_batch = np.concatenate(
+                    [previous_file['x_int'], x_int_batch],
+                    axis=0
+                )
+                x_cat_batch = np.concatenate(
+                    [previous_file['x_cat'], x_cat_batch],
+                    axis=0
+                )
+                y_batch = np.concatenate([previous_file['y'], y_batch], axis=0)
+                previous_file = None
+
+            if x_int_batch.shape[0] != batch_size:
+                raise ValueError('should not happen')
+
+            yield _transform_features(x_int_batch, x_cat_batch, y_batch, max_ind_range)
+
+            batch_start_idx += missing_samples
+        if batch_start_idx != samples_in_file:
+            current_slice = slice(batch_start_idx, samples_in_file)
+            if previous_file is not None:
+                previous_file = {
+                    'x_int' : np.concatenate(
+                        [previous_file['x_int'], x_int[current_slice]],
+                        axis=0
+                    ),
+                    'x_cat' : np.concatenate(
+                        [previous_file['x_cat'], x_cat[current_slice]],
+                        axis=0
+                    ),
+                    'y' : np.concatenate([previous_file['y'], y[current_slice]], axis=0)
+                }
+            else:
+                previous_file = {
+                    'x_int' : x_int[current_slice],
+                    'x_cat' : x_cat[current_slice],
+                    'y' : y[current_slice]
+                }
+
+    if not drop_last:
+        yield _transform_features(
+            previous_file['x_int'],
+            previous_file['x_cat'],
+            previous_file['y'],
+            max_ind_range
+        )
+
+
+def _test():
+    generator = _batch_generator(
+        data_filename='day',
+        data_directory='./input',
+        days=range(23),
+        split="train",
+        batch_size=2048,
+        drop_last=True,
+        max_ind_range=-1
+    )
+    t1 = time.time()
+    for x_int, lS_o, x_cat, y in generator:
+        t2 = time.time()
+        time_diff = t2 - t1
+        t1 = t2
+        print(
+            "time {} x_int.shape: {} lS_o.shape: {} x_cat.shape: {} y.shape: {}".format(
+                time_diff, x_int.shape, lS_o.shape, x_cat.shape, y.shape
+            )
+        )
+
+
+class CriteoBinDataset(Dataset):
+    """Binary version of criteo dataset."""
+
+    def __init__(self, data_file, counts_file,
+                 batch_size=1, max_ind_range=-1, bytes_per_feature=4):
+        # dataset
+        self.tar_fea = 1   # single target
+        self.den_fea = 13  # 13 dense  features
+        self.spa_fea = 26  # 26 sparse features
+        self.tad_fea = self.tar_fea + self.den_fea
+        self.tot_fea = self.tad_fea + self.spa_fea
+
+        self.batch_size = batch_size
+        self.max_ind_range = max_ind_range
+        self.bytes_per_entry = (bytes_per_feature * self.tot_fea * batch_size)
+
+        data_file_size = os.path.getsize(data_file)
+        self.num_entries = math.ceil(data_file_size / self.bytes_per_entry)
+
+        bytes_per_sample = bytes_per_feature * self.tot_fea
+        if ext_dist.my_size > 1:
+            self.bytes_per_rank = self.bytes_per_entry // ext_dist.my_size
+        else:
+            self.bytes_per_rank = self.bytes_per_entry
+
+        if ext_dist.my_size > 1 and self.num_entries * self.bytes_per_entry > data_file_size:
+            last_batch = (data_file_size % self.bytes_per_entry) // bytes_per_sample
+            self.bytes_last_batch = last_batch // ext_dist.my_size * bytes_per_sample
+        else:
+            self.bytes_last_batch = self.bytes_per_rank
+
+        if self.bytes_last_batch == 0:
+            self.num_entries = self.num_entries - 1
+            self.bytes_last_batch = self.bytes_per_rank
+
+        print('data file:', data_file, 'number of batches:', self.num_entries)
+        self.file = open(data_file, 'rb')
+
+        with np.load(counts_file) as data:
+            self.counts = data["counts"]
+
+        # hardcoded for now
+        self.m_den = 13
+
+    def __len__(self):
+        return self.num_entries
+
+    def __getitem__(self, idx):
+        my_rank = ext_dist.dist.get_rank() if ext_dist.my_size > 1 else 0
+        rank_size = self.bytes_last_batch if idx == (self.num_entries - 1) else self.bytes_per_rank 
+        if ext_dist.my_size > 1:
+            self.file.seek(idx * self.bytes_per_entry + rank_size * my_rank, 0)
+        else:
+            self.file.seek(idx * self.bytes_per_entry, 0)
+        raw_data = self.file.read(rank_size)
+        array = np.frombuffer(raw_data, dtype=np.int32)
+        tensor = torch.from_numpy(array).view((-1, self.tot_fea))
+
+        return _transform_features(x_int_batch=tensor[:, 1:14],
+                                   x_cat_batch=tensor[:, 14:],
+                                   y_batch=tensor[:, 0],
+                                   max_ind_range=self.max_ind_range,
+                                   flag_input_torch_tensor=True)
+
+    def __del__(self):
+        self.file.close()
+
+
+def numpy_to_binary(input_files, output_file_path, split='train'):
+    """Convert the data to a binary format to be read with CriteoBinDataset."""
+
+    # WARNING - both categorical and numerical data must fit into int32 for
+    # the following code to work correctly
+
+    with open(output_file_path, 'wb') as output_file:
+        if split == 'train':
+            for input_file in input_files:
+                print('Processing file: ', input_file)
+
+                np_data = np.load(input_file)
+                np_data = np.concatenate([np_data['y'].reshape(-1, 1),
+                                          np_data['X_int'],
+                                          np_data['X_cat']], axis=1)
+                np_data = np_data.astype(np.int32)
+
+                output_file.write(np_data.tobytes())
+        else:
+            assert len(input_files) == 1
+            np_data = np.load(input_files[0])
+            np_data = np.concatenate([np_data['y'].reshape(-1, 1),
+                                      np_data['X_int'],
+                                      np_data['X_cat']], axis=1)
+            np_data = np_data.astype(np.int32)
+
+            samples_in_file = np_data.shape[0]
+            midpoint = int(np.ceil(samples_in_file / 2.))
+            if split == "test":
+                begin = 0
+                end = midpoint
+            elif split == "val":
+                begin = midpoint
+                end = samples_in_file
+            else:
+                raise ValueError('Unknown split value: ', split)
+
+            output_file.write(np_data[begin:end].tobytes())
+
+
+def _preprocess(args):
+    train_files = ['{}_{}_reordered.npz'.format(args.input_data_prefix, day) for
+                   day in range(0, 23)]
+
+    test_valid_file = args.input_data_prefix + '_23_reordered.npz'
+
+    os.makedirs(args.output_directory, exist_ok=True)
+    for split in ['train', 'val', 'test']:
+        print('Running preprocessing for split =', split)
+
+        output_file = os.path.join(args.output_directory,
+                                   '{}_data.bin'.format(split))
+
+        input_files = train_files if split == 'train' else [test_valid_file]
+        numpy_to_binary(input_files=input_files,
+                        output_file_path=output_file,
+                        split=split)
+
+
+def _test_bin():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_directory', required=True)
+    parser.add_argument('--input_data_prefix', required=True)
+    parser.add_argument('--split', choices=['train', 'test', 'val'],
+                        required=True)
+    args = parser.parse_args()
+
+    _preprocess(args)
+
+    binary_data_file = os.path.join(args.output_directory,
+                                    '{}_data.bin'.format(args.split))
+
+    counts_file = os.path.join(args.output_directory, 'day_fea_count.npz')
+    dataset_binary = CriteoBinDataset(data_file=binary_data_file,
+                                            counts_file=counts_file,
+                                            batch_size=2048,)
+    from models.recommendation.pytorch.memrec.product.memrec_data import CriteoDataset 
+    from models.recommendation.pytorch.memrec.product.memrec_data import collate_wrapper_criteo_offset as collate_wrapper_criteo
+
+    binary_loader = torch.utils.data.DataLoader(
+        dataset_binary,
+        batch_size=None,
+        shuffle=False,
+        num_workers=0,
+        collate_fn=None,
+        pin_memory=False,
+        drop_last=False,
+    )
+
+    original_dataset = CriteoDataset(
+        dataset='terabyte',
+        max_ind_range=10 * 1000 * 1000,
+        sub_sample_rate=1,
+        randomize=True,
+        split=args.split,
+        raw_path=args.input_data_prefix,
+        pro_data='dummy_string',
+        memory_map=True
+    )
+
+    original_loader = torch.utils.data.DataLoader(
+        original_dataset,
+        batch_size=2048,
+        shuffle=False,
+        num_workers=0,
+        collate_fn=collate_wrapper_criteo,
+        pin_memory=False,
+        drop_last=False,
+    )
+
+    assert len(dataset_binary) == len(original_loader)
+    for i, (old_batch, new_batch) in tqdm(enumerate(zip(original_loader,
+                                                        binary_loader)),
+                                          total=len(dataset_binary)):
+
+        for j in range(len(new_batch)):
+            if not np.array_equal(old_batch[j], new_batch[j]):
+                raise ValueError('FAILED: Datasets not equal')
+        if i > len(dataset_binary):
+            break
+    print('PASSED')
+
+
+if __name__ == '__main__':
+    _test()
+    _test_bin()
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/data_utils.py b/models/recommendation/pytorch/memrec_dlrm/product/data_utils.py
new file mode 100644
index 000000000..bf76dfffa
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/data_utils.py
@@ -0,0 +1,1292 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+# Description: generate inputs and targets for the DLRM benchmark
+#
+# Utility function(s) to download and pre-process public data sets
+#   - Criteo Kaggle Display Advertising Challenge Dataset
+#     https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
+#   - Criteo Terabyte Dataset
+#     https://labs.criteo.com/2013/12/download-terabyte-click-logs
+#
+# After downloading dataset, run:
+#   getCriteoAdData(
+#       datafile="<path-to-train.txt>",
+#       o_filename=kaggleAdDisplayChallenge_processed.npz,
+#       max_ind_range=-1,
+#       sub_sample_rate=0.0,
+#       days=7,
+#       data_split='train',
+#       randomize='total',
+#       criteo_kaggle=True,
+#       memory_map=False
+#   )
+#   getCriteoAdData(
+#       datafile="<path-to-day_{0,...,23}>",
+#       o_filename=terabyte_processed.npz,
+#       max_ind_range=-1,
+#       sub_sample_rate=0.0,
+#       days=24,
+#       data_split='train',
+#       randomize='total',
+#       criteo_kaggle=False,
+#       memory_map=False
+#   )
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import sys
+# import os
+from os import path
+from multiprocessing import Process, Manager
+# import io
+# from io import StringIO
+# import collections as coll
+
+import numpy as np
+
+
+def convertUStringToDistinctIntsDict(mat, convertDicts, counts):
+    # Converts matrix of unicode strings into distinct integers.
+    #
+    # Inputs:
+    #     mat (np.array): array of unicode strings to convert
+    #     convertDicts (list): dictionary for each column
+    #     counts (list): number of different categories in each column
+    #
+    # Outputs:
+    #     out (np.array): array of output integers
+    #     convertDicts (list): dictionary for each column
+    #     counts (list): number of different categories in each column
+
+    # check if convertDicts and counts match correct length of mat
+    if len(convertDicts) != mat.shape[1] or len(counts) != mat.shape[1]:
+        print("Length of convertDicts or counts does not match input shape")
+        print("Generating convertDicts and counts...")
+
+        convertDicts = [{} for _ in range(mat.shape[1])]
+        counts = [0 for _ in range(mat.shape[1])]
+
+    # initialize output
+    out = np.zeros(mat.shape)
+
+    for j in range(mat.shape[1]):
+        for i in range(mat.shape[0]):
+            # add to convertDict and increment count
+            if mat[i, j] not in convertDicts[j]:
+                convertDicts[j][mat[i, j]] = counts[j]
+                counts[j] += 1
+            out[i, j] = convertDicts[j][mat[i, j]]
+
+    return out, convertDicts, counts
+
+
+def convertUStringToDistinctIntsUnique(mat, mat_uni, counts):
+    # mat is an array of 0,...,# samples, with each being 26 categorical features
+
+    # check if mat_unique and counts match correct length of mat
+    if len(mat_uni) != mat.shape[1] or len(counts) != mat.shape[1]:
+        print("Length of mat_unique or counts does not match input shape")
+        print("Generating mat_unique and counts...")
+
+        mat_uni = [np.array([]) for _ in range(mat.shape[1])]
+        counts = [0 for _ in range(mat.shape[1])]
+
+    # initialize output
+    out = np.zeros(mat.shape)
+    ind_map = [np.array([]) for _ in range(mat.shape[1])]
+
+    # find out and assign unique ids to features
+    for j in range(mat.shape[1]):
+        m = mat_uni[j].size
+        mat_concat = np.concatenate((mat_uni[j], mat[:, j]))
+        mat_uni[j], ind_map[j] = np.unique(mat_concat, return_inverse=True)
+        out[:, j] = ind_map[j][m:]
+        counts[j] = mat_uni[j].size
+
+    return out, mat_uni, counts
+
+
+def processCriteoAdData(d_path, d_file, npzfile, i, convertDicts, pre_comp_counts):
+    # Process Kaggle Display Advertising Challenge or Terabyte Dataset
+    # by converting unicode strings in X_cat to integers and
+    # converting negative integer values in X_int.
+    #
+    # Loads data in the form "{kaggle|terabyte}_day_i.npz" where i is the day.
+    #
+    # Inputs:
+    #   d_path (str): path for {kaggle|terabyte}_day_i.npz files
+    #   i (int): splits in the dataset (typically 0 to 7 or 0 to 24)
+
+    # process data if not all files exist
+    filename_i = npzfile + "_{0}_processed.npz".format(i)
+
+    if path.exists(filename_i):
+        print("Using existing " + filename_i, end="\n")
+    else:
+        print("Not existing " + filename_i)
+        with np.load(npzfile + "_{0}.npz".format(i)) as data:
+            # categorical features
+            '''
+            # Approach 1a: using empty dictionaries
+            X_cat, convertDicts, counts = convertUStringToDistinctIntsDict(
+                data["X_cat"], convertDicts, counts
+            )
+            '''
+            '''
+            # Approach 1b: using empty np.unique
+            X_cat, convertDicts, counts = convertUStringToDistinctIntsUnique(
+                data["X_cat"], convertDicts, counts
+            )
+            '''
+            # Approach 2a: using pre-computed dictionaries
+            X_cat_t = np.zeros(data["X_cat_t"].shape)
+            for j in range(26):
+                for k, x in enumerate(data["X_cat_t"][j, :]):
+                    X_cat_t[j, k] = convertDicts[j][x]
+            # continuous features
+            X_int = data["X_int"]
+            X_int[X_int < 0] = 0
+            # targets
+            y = data["y"]
+
+        np.savez_compressed(
+            filename_i,
+            # X_cat = X_cat,
+            X_cat=np.transpose(X_cat_t),  # transpose of the data
+            X_int=X_int,
+            y=y,
+        )
+        print("Processed " + filename_i, end="\n")
+    # sanity check (applicable only if counts have been pre-computed & are re-computed)
+    # for j in range(26):
+    #    if pre_comp_counts[j] != counts[j]:
+    #        sys.exit("ERROR: Sanity check on counts has failed")
+    # print("\nSanity check on counts passed")
+
+    return
+
+
+def concatCriteoAdData(
+        d_path,
+        d_file,
+        npzfile,
+        trafile,
+        days,
+        data_split,
+        randomize,
+        total_per_file,
+        total_count,
+        memory_map,
+        o_filename
+):
+    # Concatenates different days and saves the result.
+    #
+    # Inputs:
+    #   days (int): total number of days in the dataset (typically 7 or 24)
+    #   d_path (str): path for {kaggle|terabyte}_day_i.npz files
+    #   o_filename (str): output file name
+    #
+    # Output:
+    #   o_file (str): output file path
+
+    if memory_map:
+        # dataset break up per fea
+        # tar_fea = 1   # single target
+        den_fea = 13  # 13 dense  features
+        spa_fea = 26  # 26 sparse features
+        # tad_fea = tar_fea + den_fea
+        # tot_fea = tad_fea + spa_fea
+        # create offset per file
+        offset_per_file = np.array([0] + [x for x in total_per_file])
+        for i in range(days):
+            offset_per_file[i + 1] += offset_per_file[i]
+
+        '''
+        # Approach 1, 2 and 3 use indices, while Approach 4 does not use them
+        # create indices
+        indices = np.arange(total_count)
+        if data_split == "none":
+            if randomize == "total":
+                indices = np.random.permutation(indices)
+        else:
+            indices = np.array_split(indices, offset_per_file[1:-1])
+
+            # randomize train data (per day)
+            if randomize == "day":  # or randomize == "total":
+                for i in range(len(indices) - 1):
+                    indices[i] = np.random.permutation(indices[i])
+                print("Randomized indices per day ...")
+
+            train_indices = np.concatenate(indices[:-1])
+            test_indices = indices[-1]
+
+            # randomize train data (across days)
+            if randomize == "total":
+                train_indices = np.random.permutation(train_indices)
+                print("Randomized indices across days ...")
+
+            indices = np.concatenate((train_indices, test_indices))
+        # no reordering
+        # indices = np.arange(total_count)
+        '''
+        '''
+        # Approach 1: simple and slow (no grouping is used)
+        # check if data already exists
+        recreate_flag = False
+        for j in range(tot_fea):
+            filename_j = trafile + "_{0}_reordered.npy".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((total_count))
+            for j in range(tot_fea):
+                filename_j = trafile + "_{0}_reordered".format(j)
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat_t = np.transpose(data["X_cat"])
+                    X_int_t = np.transpose(data["X_int"])
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                # print(filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #     + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    if j < tar_fea:
+                        fj[indices[start:end]] = y
+                    elif tar_fea <= j and j < tad_fea:
+                        fj[indices[start:end]] = X_int_t[j - tar_fea, :]
+                    else:
+                        fj[indices[start:end]] = X_cat_t[j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing " + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                size = total_per_file[i]
+                X_int_t = np.zeros((den_fea, size))
+                X_cat_t = np.zeros((spa_fea, size))
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                print("Creating " + filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #     + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r')
+                    if j < tar_fea:
+                        y = fj[start:end]
+                    elif tar_fea <= j and j < tad_fea:
+                        X_int_t[j - tar_fea, :] = fj[start:end]
+                    else:
+                        X_cat_t[j - tad_fea, :] = fj[start:end]
+                    del fj
+
+                np.savez_compressed(
+                    filename_i,
+                    X_cat=np.transpose(X_cat_t),  # transpose of the data
+                    X_int=np.transpose(X_int_t),  # transpose of the data
+                    y=y,
+                )
+        else:
+            print("Reordered day files already exist, skipping ...")
+        '''
+        '''
+        # Approach 2: group days
+        # check if data already exists
+        recreate_flag = False
+        for j in range(tot_fea):
+            filename_j = trafile + "_{0}_reordered.npy".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((total_count))
+            for j in range(tot_fea):
+                filename_j = trafile + "_{0}_reordered".format(j)
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            group_day = 3  # e.g. 8, 4 or 3
+            group_num = days // group_day
+            file_group = [i*group_day for i in range(group_num)] + [days]
+            for ii in range(group_num):
+                # for last may be group_size != group_num, therefore reset it below
+                group_size = file_group[ii + 1] - file_group[ii]
+                X_cat_t = [0]*group_size
+                X_int_t = [0]*group_size
+                y = [0]*group_size
+                start = [0]*group_size
+                end  = [0]*group_size
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                    # setup start and end ranges
+                    start[ig] = offset_per_file[i]
+                    end[ig] = offset_per_file[i + 1]
+                    # print(filename_i)
+                    # load a group of files
+                    with np.load(filename_i) as data:
+                        X_cat_t[ig] = np.transpose(data["X_cat"])
+                        X_int_t[ig] = np.transpose(data["X_int"])
+                        y[ig] = data["y"]
+                    # sanity check
+                    if total_per_file[i] != len(y[ig]):
+                        sys.exit("ERROR: sanity check on number of samples failed")
+                # print("start=" + str(start) + " end=" + str(end)
+                #  + " diff=" + str(end[ig]-start[ig]) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    for ig in range(group_size):
+                        if j < tar_fea:
+                            fj[indices[start[ig]:end[ig]]] = y[ig]
+                        elif tar_fea <= j and j < tad_fea:
+                            fj[indices[start[ig]:end[ig]]] = X_int_t[ig][j - tar_fea, :]
+                        else:
+                            fj[indices[start[ig]:end[ig]]] = X_cat_t[ig][j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing " + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for ii in range(group_num):
+                # for last may be group_size != group_num, therefore reset it below
+                group_size = file_group[ii + 1] - file_group[ii]
+                X_cat_t= []; X_int_t = []
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    X_int_t.append(np.zeros((den_fea, total_per_file[i])))
+                    X_cat_t.append(np.zeros((spa_fea, total_per_file[i])))
+                y = [0]*group_size
+                start = [0]*group_size
+                end  = [0]*group_size
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r')
+                    # load a group of files
+                    for ig in range(group_size):
+                        i = file_group[ii] + ig
+                        # setup start and end ranges
+                        start[ig] = offset_per_file[i]
+                        end[ig] = offset_per_file[i + 1]
+                        # load data for the group of files
+                        if j < tar_fea:
+                            y[ig] = fj[start[ig]:end[ig]]
+                        elif tar_fea <= j and j < tad_fea:
+                            X_int_t[ig][j - tar_fea, :] = fj[start[ig]:end[ig]]
+                        else:
+                            X_cat_t[ig][j - tad_fea, :] = fj[start[ig]:end[ig]]
+                    del fj
+
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                    print("Creating " + filename_i)
+                    np.savez_compressed(
+                        filename_i,
+                        X_cat=np.transpose(X_cat_t[ig]),  # transpose of the data
+                        X_int=np.transpose(X_int_t[ig]),  # transpose of the data
+                        y=y[ig],
+                    )
+        else:
+            print("Reordered day files already exist, skipping ...")
+        '''
+        '''
+        # Approach 3: group features
+        # check if data already exists
+        group_fea = 5  # e.g. 8, 5 or 4
+        group_num = tot_fea // group_fea
+        if tot_fea % group_fea != 0:  # sanity check
+            sys.exit("ERROR: the group_fea must divided tot_fea evenly.")
+        recreate_flag = False
+        for jn in range(group_num):
+            filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                jn, group_fea
+            )
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((group_fea, total_count))
+            for jn in range(group_num):
+                filename_j = trafile + "_{0}_reordered{1}".format(
+                    jn, group_fea
+                )
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat_t = np.transpose(data["X_cat"])
+                    X_int_t = np.transpose(data["X_int"])
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                # print(filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #      + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for jn in range(group_num):
+                    filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                        jn, group_fea
+                    )
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    for jg in range(group_fea):
+                        j = jn * group_fea + jg
+                        # print("j=" + str(j) + " jn=" + str(jn) + " jg=" + str(jg))
+                        if j < tar_fea:
+                            fj[jg, indices[start:end]] = y
+                        elif tar_fea <= j and j < tad_fea:
+                            fj[jg, indices[start:end]] = X_int_t[j - tar_fea, :]
+                        else:
+                            fj[jg, indices[start:end]] = X_cat_t[j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing" + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                size = total_per_file[i]
+                X_int_t = np.zeros((den_fea, size))
+                X_cat_t = np.zeros((spa_fea, size))
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                print("Creating " + filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #      + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for jn in range(group_num):
+                    filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                        jn, group_fea
+                    )
+                    fj = np.load(filename_j, mmap_mode='r')
+                    for jg in range(group_fea):
+                        j = jn * group_fea + jg
+                        # print("j=" + str(j) + " jn=" + str(jn) + " jg=" + str(jg))
+                        if j < tar_fea:
+                            y = fj[jg, start:end]
+                        elif tar_fea <= j and j < tad_fea:
+                            X_int_t[j - tar_fea, :] = fj[jg, start:end]
+                        else:
+                            X_cat_t[j - tad_fea, :] = fj[jg, start:end]
+                    del fj
+
+                np.savez_compressed(
+                    filename_i,
+                    X_cat=np.transpose(X_cat_t),  # transpose of the data
+                    X_int=np.transpose(X_int_t),  # transpose of the data
+                    y=y,
+                )
+
+        else:
+            print("Reordered day files already exist, skipping ...")
+        '''
+
+        # Approach 4: Fisher-Yates-Rao (FYR) shuffle algorithm
+        # 1st pass of FYR shuffle
+        # check if data already exists
+        recreate_flag = False
+        for j in range(days):
+            filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+            filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+            filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+            if (
+                path.exists(filename_j_y)
+                and path.exists(filename_j_d)
+                and path.exists(filename_j_s)
+            ):
+                print(
+                    "Using existing\n"
+                    + filename_j_y + "\n"
+                    + filename_j_d + "\n"
+                    + filename_j_s
+                )
+            else:
+                recreate_flag = True
+        # reorder across buckets using sampling
+        if recreate_flag:
+            # init intermediate files (.npy appended automatically)
+            for j in range(days):
+                filename_j_y = npzfile + "_{0}_intermediate_y".format(j)
+                filename_j_d = npzfile + "_{0}_intermediate_d".format(j)
+                filename_j_s = npzfile + "_{0}_intermediate_s".format(j)
+                np.save(filename_j_y, np.zeros((total_per_file[j])))
+                np.save(filename_j_d, np.zeros((total_per_file[j], den_fea)))
+                np.save(filename_j_s, np.zeros((total_per_file[j], spa_fea)))
+            # start processing files
+            total_counter = [0] * days
+            for i in range(days):
+                filename_i = npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat = data["X_cat"]
+                    X_int = data["X_int"]
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # debug prints
+                print("Reordering (1st pass) " + filename_i)
+
+                # create buckets using sampling of random ints
+                # from (discrete) uniform distribution
+                buckets = []
+                for _j in range(days):
+                    buckets.append([])
+                counter = [0] * days
+                days_to_sample = days if data_split == "none" else days - 1
+                if randomize == "total":
+                    rand_u = np.random.randint(low=0, high=days_to_sample, size=size)
+                    for k in range(size):
+                        # sample and make sure elements per buckets do not overflow
+                        if data_split == "none" or i < days - 1:
+                            # choose bucket
+                            p = rand_u[k]
+                            # retry of the bucket is full
+                            while total_counter[p] + counter[p] >= total_per_file[p]:
+                                p = np.random.randint(low=0, high=days_to_sample)
+                        else:  # preserve the last day/bucket if needed
+                            p = i
+                        buckets[p].append(k)
+                        counter[p] += 1
+                else:  # randomize is day or none
+                    for k in range(size):
+                        # do not sample, preserve the data in this bucket
+                        p = i
+                        buckets[p].append(k)
+                        counter[p] += 1
+
+                # sanity check
+                if np.sum(counter) != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # debug prints
+                # print(counter)
+                # print(str(np.sum(counter)) + " = " + str(size))
+                # print([len(x) for x in buckets])
+                # print(total_counter)
+
+                # partially feel the buckets
+                for j in range(days):
+                    filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+                    filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+                    filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+                    start = total_counter[j]
+                    end = total_counter[j] + counter[j]
+                    # target buckets
+                    fj_y = np.load(filename_j_y, mmap_mode='r+')
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_y[start:end].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_y[start:end] = y[buckets[j]]
+                    del fj_y
+                    # dense buckets
+                    fj_d = np.load(filename_j_d, mmap_mode='r+')
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_d[start:end, :].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_d[start:end, :] = X_int[buckets[j], :]
+                    del fj_d
+                    # sparse buckets
+                    fj_s = np.load(filename_j_s, mmap_mode='r+')
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_s[start:end, :].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_s[start:end, :] = X_cat[buckets[j], :]
+                    del fj_s
+                    # update counters for next step
+                    total_counter[j] += counter[j]
+
+        # 2nd pass of FYR shuffle
+        # check if data already exists
+        for j in range(days):
+            filename_j = npzfile + "_{0}_reordered.npz".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # reorder within buckets
+        if recreate_flag:
+            for j in range(days):
+                filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+                filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+                filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+                fj_y = np.load(filename_j_y)
+                fj_d = np.load(filename_j_d)
+                fj_s = np.load(filename_j_s)
+
+                indices = range(total_per_file[j])
+                if randomize == "day" or randomize == "total":
+                    if data_split == "none" or j < days - 1:
+                        indices = np.random.permutation(range(total_per_file[j]))
+
+                filename_r = npzfile + "_{0}_reordered.npz".format(j)
+                print("Reordering (2nd pass) " + filename_r)
+                np.savez_compressed(
+                    filename_r,
+                    X_cat=fj_s[indices, :],
+                    X_int=fj_d[indices, :],
+                    y=fj_y[indices],
+                )
+
+        '''
+        # sanity check (under no reordering norms should be zero)
+        for i in range(days):
+            filename_i_o = npzfile + "_{0}_processed.npz".format(i)
+            print(filename_i_o)
+            with np.load(filename_i_o) as data_original:
+                X_cat_o = data_original["X_cat"]
+                X_int_o = data_original["X_int"]
+                y_o = data_original["y"]
+            filename_i_r = npzfile + "_{0}_reordered.npz".format(i)
+            print(filename_i_r)
+            with np.load(filename_i_r) as data_reordered:
+                X_cat_r = data_reordered["X_cat"]
+                X_int_r = data_reordered["X_int"]
+                y_r = data_reordered["y"]
+            print(np.linalg.norm(y_o - y_r))
+            print(np.linalg.norm(X_int_o - X_int_r))
+            print(np.linalg.norm(X_cat_o - X_cat_r))
+        '''
+
+    else:
+        print("Concatenating multiple days into %s.npz file" % str(d_path + o_filename))
+
+        # load and concatenate data
+        for i in range(days):
+            filename_i = npzfile + "_{0}_processed.npz".format(i)
+            with np.load(filename_i) as data:
+                if i == 0:
+                    X_cat = data["X_cat"]
+                    X_int = data["X_int"]
+                    y = data["y"]
+                else:
+                    X_cat = np.concatenate((X_cat, data["X_cat"]))
+                    X_int = np.concatenate((X_int, data["X_int"]))
+                    y = np.concatenate((y, data["y"]))
+            print("Loaded day:", i, "y = 1:", len(y[y == 1]), "y = 0:", len(y[y == 0]))
+
+        with np.load(d_path + d_file + "_fea_count.npz") as data:
+            counts = data["counts"]
+        print("Loaded counts!")
+
+        np.savez_compressed(
+            d_path + o_filename + ".npz",
+            X_cat=X_cat,
+            X_int=X_int,
+            y=y,
+            counts=counts,
+        )
+
+    return d_path + o_filename + ".npz"
+
+
+def transformCriteoAdData(X_cat, X_int, y, days, data_split, randomize, total_per_file):
+    # Transforms Criteo Kaggle or terabyte data by applying log transformation
+    # on dense features and converting everything to appropriate tensors.
+    #
+    # Inputs:
+    #     X_cat (ndarray): array of integers corresponding to preprocessed
+    #                      categorical features
+    #     X_int (ndarray): array of integers corresponding to dense features
+    #     y (ndarray):     array of bool corresponding to labels
+    #     data_split(str): flag for splitting dataset into training/validation/test
+    #                      sets
+    #     randomize (str): determines randomization scheme
+    #         "none": no randomization
+    #         "day": randomizes each day"s data (only works if split = True)
+    #         "total": randomizes total dataset
+    #
+    # Outputs:
+    #     if split:
+    #         X_cat_train (tensor): sparse features for training set
+    #         X_int_train (tensor): dense features for training set
+    #         y_train (tensor): labels for training set
+    #         X_cat_val (tensor): sparse features for validation set
+    #         X_int_val (tensor): dense features for validation set
+    #         y_val (tensor): labels for validation set
+    #         X_cat_test (tensor): sparse features for test set
+    #         X_int_test (tensor): dense features for test set
+    #         y_test (tensor): labels for test set
+    #     else:
+    #         X_cat (tensor): sparse features
+    #         X_int (tensor): dense features
+    #         y (tensor): label
+
+    # define initial set of indices
+    indices = np.arange(len(y))
+
+    # create offset per file
+    offset_per_file = np.array([0] + [x for x in total_per_file])
+    for i in range(days):
+        offset_per_file[i + 1] += offset_per_file[i]
+
+    # split dataset
+    if data_split == 'train':
+        indices = np.array_split(indices, offset_per_file[1:-1])
+
+        # randomize train data (per day)
+        if randomize == "day":  # or randomize == "total":
+            for i in range(len(indices) - 1):
+                indices[i] = np.random.permutation(indices[i])
+            print("Randomized indices per day ...")
+
+        train_indices = np.concatenate(indices[:-1])
+        test_indices = indices[-1]
+        test_indices, val_indices = np.array_split(test_indices, 2)
+
+        print("Defined training and testing indices...")
+
+        # randomize train data (across days)
+        if randomize == "total":
+            train_indices = np.random.permutation(train_indices)
+            print("Randomized indices across days ...")
+
+        # indices = np.concatenate((train_indices, test_indices))
+
+        # create training, validation, and test sets
+        X_cat_train = X_cat[train_indices]
+        X_int_train = X_int[train_indices]
+        y_train = y[train_indices]
+
+        X_cat_val = X_cat[val_indices]
+        X_int_val = X_int[val_indices]
+        y_val = y[val_indices]
+
+        X_cat_test = X_cat[test_indices]
+        X_int_test = X_int[test_indices]
+        y_test = y[test_indices]
+
+        print("Split data according to indices...")
+
+        X_cat_train = X_cat_train.astype(np.long)
+        X_int_train = np.log(X_int_train.astype(np.float32) + 1)
+        y_train = y_train.astype(np.float32)
+
+        X_cat_val = X_cat_val.astype(np.long)
+        X_int_val = np.log(X_int_val.astype(np.float32) + 1)
+        y_val = y_val.astype(np.float32)
+
+        X_cat_test = X_cat_test.astype(np.long)
+        X_int_test = np.log(X_int_test.astype(np.float32) + 1)
+        y_test = y_test.astype(np.float32)
+
+        print("Converted to tensors...done!")
+
+        return (
+            X_cat_train,
+            X_int_train,
+            y_train,
+            X_cat_val,
+            X_int_val,
+            y_val,
+            X_cat_test,
+            X_int_test,
+            y_test,
+        )
+
+    else:
+
+        # randomize data
+        if randomize == "total":
+            indices = np.random.permutation(indices)
+            print("Randomized indices...")
+
+        X_cat = X_cat[indices].astype(np.long)
+        X_int = np.log(X_int[indices].astype(np.float32) + 1)
+        y = y[indices].astype(np.float32)
+
+        print("Converted to tensors...done!")
+
+        return (X_cat, X_int, y, [], [], [], [], [], [])
+
+
+def getCriteoAdData(
+        datafile,
+        o_filename,
+        max_ind_range=-1,
+        sub_sample_rate=0.0,
+        days=7,
+        data_split='train',
+        randomize='total',
+        criteo_kaggle=True,
+        memory_map=False,
+        dataset_multiprocessing=False,
+):
+    # Passes through entire dataset and defines dictionaries for categorical
+    # features and determines the number of total categories.
+    #
+    # Inputs:
+    #    datafile : path to downloaded raw data file
+    #    o_filename (str): saves results under o_filename if filename is not ""
+    #
+    # Output:
+    #   o_file (str): output file path
+
+    #split the datafile into path and filename
+    lstr = datafile.split("/")
+    d_path = "/".join(lstr[0:-1]) + "/"
+    d_file = lstr[-1].split(".")[0] if criteo_kaggle else lstr[-1]
+    npzfile = d_path + ((d_file + "_day") if criteo_kaggle else d_file)
+    trafile = d_path + ((d_file + "_fea") if criteo_kaggle else "fea")
+
+    # count number of datapoints in training set
+    total_file = d_path + d_file + "_day_count.npz"
+    if path.exists(total_file):
+        with np.load(total_file) as data:
+            total_per_file = list(data["total_per_file"])
+        total_count = np.sum(total_per_file)
+        print("Skipping counts per file (already exist)")
+    else:
+        total_count = 0
+        total_per_file = []
+        if criteo_kaggle:
+            # WARNING: The raw data consists of a single train.txt file
+            # Each line in the file is a sample, consisting of 13 continuous and
+            # 26 categorical features (an extra space indicates that feature is
+            # missing and will be interpreted as 0).
+            if path.exists(datafile):
+                print("Reading data from path=%s" % (datafile))
+                with open(str(datafile)) as f:
+                    for _ in f:
+                        total_count += 1
+                total_per_file.append(total_count)
+                # reset total per file due to split
+                num_data_per_split, extras = divmod(total_count, days)
+                total_per_file = [num_data_per_split] * days
+                for j in range(extras):
+                    total_per_file[j] += 1
+                # split into days (simplifies code later on)
+                file_id = 0
+                boundary = total_per_file[file_id]
+                nf = open(npzfile + "_" + str(file_id), "w")
+                with open(str(datafile)) as f:
+                    for j, line in enumerate(f):
+                        if j == boundary:
+                            nf.close()
+                            file_id += 1
+                            nf = open(npzfile + "_" + str(file_id), "w")
+                            boundary += total_per_file[file_id]
+                        nf.write(line)
+                nf.close()
+            else:
+                sys.exit("ERROR: Criteo Kaggle Display Ad Challenge Dataset path is invalid; please download from https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset")
+        else:
+            # WARNING: The raw data consist of day_0.gz,... ,day_23.gz text files
+            # Each line in the file is a sample, consisting of 13 continuous and
+            # 26 categorical features (an extra space indicates that feature is
+            # missing and will be interpreted as 0).
+            for i in range(days):
+                datafile_i = datafile + "_" + str(i)  # + ".gz"
+                if path.exists(str(datafile_i)):
+                    print("Reading data from path=%s" % (str(datafile_i)))
+                    # file day_<number>
+                    total_per_file_count = 0
+                    with open(str(datafile_i)) as f:
+                        for _ in f:
+                            total_per_file_count += 1
+                    total_per_file.append(total_per_file_count)
+                    total_count += total_per_file_count
+                else:
+                    sys.exit("ERROR: Criteo Terabyte Dataset path is invalid; please download from https://labs.criteo.com/2013/12/download-terabyte-click-logs")
+
+    # process a file worth of data and reinitialize data
+    # note that a file main contain a single or multiple splits
+    def process_one_file(
+            datfile,
+            npzfile,
+            split,
+            num_data_in_split,
+            dataset_multiprocessing,
+            convertDictsDay=None,
+            resultDay=None
+    ):
+        if dataset_multiprocessing:
+            convertDicts_day = [{} for _ in range(26)]
+
+        with open(str(datfile)) as f:
+            y = np.zeros(num_data_in_split, dtype="i4")  # 4 byte int
+            X_int = np.zeros((num_data_in_split, 13), dtype="i4")  # 4 byte int
+            X_cat = np.zeros((num_data_in_split, 26), dtype="i4")  # 4 byte int
+            if sub_sample_rate == 0.0:
+                rand_u = 1.0
+            else:
+                rand_u = np.random.uniform(low=0.0, high=1.0, size=num_data_in_split)
+
+            i = 0
+            percent = 0
+            for k, line in enumerate(f):
+                # process a line (data point)
+                line = line.split('\t')
+                # set missing values to zero
+                for j in range(len(line)):
+                    if (line[j] == '') or (line[j] == '\n'):
+                        line[j] = '0'
+                # sub-sample data by dropping zero targets, if needed
+                target = np.int32(line[0])
+                if target == 0 and \
+                   (rand_u if sub_sample_rate == 0.0 else rand_u[k]) < sub_sample_rate:
+                    continue
+
+                y[i] = target
+                X_int[i] = np.array(line[1:14], dtype=np.int32)
+                if max_ind_range > 0:
+                    X_cat[i] = np.array(
+                        list(map(lambda x: int(x, 16) % max_ind_range, line[14:])),
+                        dtype=np.int32
+                    )
+                else:
+                    X_cat[i] = np.array(
+                        list(map(lambda x: int(x, 16), line[14:])),
+                        dtype=np.int32
+                    )
+
+                # count uniques
+                if dataset_multiprocessing:
+                    for j in range(26):
+                        convertDicts_day[j][X_cat[i][j]] = 1
+                    # debug prints
+                    if float(i)/num_data_in_split*100 > percent+1:
+                        percent = int(float(i)/num_data_in_split*100)
+                        print(
+                            "Load %d/%d (%d%%) Split: %d  Label True: %d  Stored: %d"
+                            % (
+                                i,
+                                num_data_in_split,
+                                percent,
+                                split,
+                                target,
+                                y[i],
+                            ),
+                            end="\n",
+                        )
+                else:
+                    for j in range(26):
+                        convertDicts[j][X_cat[i][j]] = 1
+                    # debug prints
+                    print(
+                        "Load %d/%d  Split: %d  Label True: %d  Stored: %d"
+                        % (
+                            i,
+                            num_data_in_split,
+                            split,
+                            target,
+                            y[i],
+                        ),
+                        end="\r",
+                    )
+                i += 1
+
+            # store num_data_in_split samples or extras at the end of file
+            # count uniques
+            # X_cat_t  = np.transpose(X_cat)
+            # for j in range(26):
+            #     for x in X_cat_t[j,:]:
+            #         convertDicts[j][x] = 1
+            # store parsed
+            filename_s = npzfile + "_{0}.npz".format(split)
+            if path.exists(filename_s):
+                print("\nSkip existing " + filename_s)
+            else:
+                np.savez_compressed(
+                    filename_s,
+                    X_int=X_int[0:i, :],
+                    # X_cat=X_cat[0:i, :],
+                    X_cat_t=np.transpose(X_cat[0:i, :]),  # transpose of the data
+                    y=y[0:i],
+                )
+                print("\nSaved " + npzfile + "_{0}.npz!".format(split))
+
+        if dataset_multiprocessing:
+            resultDay[split] = i
+            convertDictsDay[split] = convertDicts_day
+            return
+        else:
+            return i
+
+    # create all splits (reuse existing files if possible)
+    recreate_flag = False
+    convertDicts = [{} for _ in range(26)]
+    # WARNING: to get reproducable sub-sampling results you must reset the seed below
+    # np.random.seed(123)
+    # in this case there is a single split in each day
+    for i in range(days):
+        npzfile_i = npzfile + "_{0}.npz".format(i)
+        npzfile_p = npzfile + "_{0}_processed.npz".format(i)
+        if path.exists(npzfile_i):
+            print("Skip existing " + npzfile_i)
+        elif path.exists(npzfile_p):
+            print("Skip existing " + npzfile_p)
+        else:
+            recreate_flag = True
+
+    if recreate_flag:
+        if dataset_multiprocessing:
+            resultDay = Manager().dict()
+            convertDictsDay = Manager().dict()
+            processes = [Process(target=process_one_file,
+                                 name="process_one_file:%i" % i,
+                                 args=(npzfile + "_{0}".format(i),
+                                       npzfile,
+                                       i,
+                                       total_per_file[i],
+                                       dataset_multiprocessing,
+                                       convertDictsDay,
+                                       resultDay,
+                                       )
+                                 ) for i in range(0, days)]
+            for process in processes:
+                process.start()
+            for process in processes:
+                process.join()
+            for day in range(days):
+                total_per_file[day] = resultDay[day]
+                print("Constructing convertDicts Split: {}".format(day))
+                convertDicts_tmp = convertDictsDay[day]
+                for i in range(26):
+                    for j in convertDicts_tmp[i]:
+                        convertDicts[i][j] = 1
+        else:
+            for i in range(days):
+                total_per_file[i] = process_one_file(
+                    npzfile + "_{0}".format(i),
+                    npzfile,
+                    i,
+                    total_per_file[i],
+                    dataset_multiprocessing,
+                )
+
+    # report and save total into a file
+    total_count = np.sum(total_per_file)
+    if not path.exists(total_file):
+        np.savez_compressed(total_file, total_per_file=total_per_file)
+    print("Total number of samples:", total_count)
+    print("Divided into days/splits:\n", total_per_file)
+
+    # dictionary files
+    counts = np.zeros(26, dtype=np.int32)
+    if recreate_flag:
+        # create dictionaries
+        for j in range(26):
+            for i, x in enumerate(convertDicts[j]):
+                convertDicts[j][x] = i
+            dict_file_j = d_path + d_file + "_fea_dict_{0}.npz".format(j)
+            if not path.exists(dict_file_j):
+                np.savez_compressed(
+                    dict_file_j,
+                    unique=np.array(list(convertDicts[j]), dtype=np.int32)
+                )
+            counts[j] = len(convertDicts[j])
+        # store (uniques and) counts
+        count_file = d_path + d_file + "_fea_count.npz"
+        if not path.exists(count_file):
+            np.savez_compressed(count_file, counts=counts)
+    else:
+        # create dictionaries (from existing files)
+        for j in range(26):
+            with np.load(d_path + d_file + "_fea_dict_{0}.npz".format(j)) as data:
+                unique = data["unique"]
+            for i, x in enumerate(unique):
+                convertDicts[j][x] = i
+        # load (uniques and) counts
+        with np.load(d_path + d_file + "_fea_count.npz") as data:
+            counts = data["counts"]
+
+    # process all splits
+    if dataset_multiprocessing:
+        processes = [Process(target=processCriteoAdData,
+                           name="processCriteoAdData:%i" % i,
+                           args=(d_path,
+                                 d_file,
+                                 npzfile,
+                                 i,
+                                 convertDicts,
+                                 counts,
+                                 )
+                           ) for i in range(0, days)]
+        for process in processes:
+            process.start()
+        for process in processes:
+            process.join()
+
+    else:
+        for i in range(days):
+            processCriteoAdData(d_path, d_file, npzfile, i, convertDicts, counts)
+
+    o_file = concatCriteoAdData(
+        d_path,
+        d_file,
+        npzfile,
+        trafile,
+        days,
+        data_split,
+        randomize,
+        total_per_file,
+        total_count,
+        memory_map,
+        o_filename
+    )
+
+    return o_file
+
+
+def loadDataset(
+        dataset,
+        max_ind_range,
+        sub_sample_rate,
+        randomize,
+        data_split,
+        raw_path="",
+        pro_data="",
+        memory_map=False
+):
+    # dataset
+    if dataset == "kaggle":
+        days = 7
+        o_filename = "kaggleAdDisplayChallenge_processed"
+    elif dataset == "terabyte":
+        days = 24
+        o_filename = "terabyte_processed"
+    else:
+        raise(ValueError("Data set option is not supported"))
+
+    # split the datafile into path and filename
+    lstr = raw_path.split("/")
+    d_path = "/".join(lstr[0:-1]) + "/"
+    d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+    npzfile = (d_file + "_day") if dataset == "kaggle" else d_file
+    # trafile = d_path + ((d_file + "_fea") if dataset == "kaggle" else "fea")
+
+    # check if pre-processed data is available
+    data_ready = True
+    if memory_map:
+        for i in range(days):
+            reo_data = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if not path.exists(str(reo_data)):
+                data_ready = False
+    else:
+        if not path.exists(str(pro_data)):
+            data_ready = False
+
+    # pre-process data if needed
+    # WARNNING: when memory mapping is used we get a collection of files
+    if data_ready:
+        print("Reading pre-processed data=%s" % (str(pro_data)))
+        file = str(pro_data)
+    else:
+        print("Reading raw data=%s" % (str(raw_path)))
+        file = getCriteoAdData(
+            raw_path,
+            o_filename,
+            max_ind_range,
+            sub_sample_rate,
+            days,
+            data_split,
+            randomize,
+            dataset == "kaggle",
+            memory_map
+        )
+
+    return file, days
+
+
+if __name__ == "__main__":
+    ### import packages ###
+    import argparse
+
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(
+        description="Preprocess Criteo dataset"
+    )
+    # model related parameters
+    parser.add_argument("--max-ind-range", type=int, default=-1)
+    parser.add_argument("--data-sub-sample-rate", type=float, default=0.0)  # in [0, 1]
+    parser.add_argument("--data-randomize", type=str, default="total")  # or day or none
+    parser.add_argument("--memory-map", action="store_true", default=False)
+    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
+    parser.add_argument("--raw-data-file", type=str, default="")
+    parser.add_argument("--processed-data-file", type=str, default="")
+    args = parser.parse_args()
+
+    loadDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "train",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map
+    )
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/extend_distributed.py b/models/recommendation/pytorch/memrec_dlrm/product/extend_distributed.py
new file mode 100644
index 000000000..66e13ad87
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/extend_distributed.py
@@ -0,0 +1,460 @@
+import os
+import builtins
+import numpy as np
+import torch
+from torch.autograd import Function
+from torch.nn.parallel import DistributedDataParallel as DDP
+import torch.distributed as dist
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+    else:
+        import torch_ccl
+except ImportError as e:
+    #print(e)
+    oneccl_bindings_for_pytorch = False
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+allgatherv_supported = False
+a2a_impl = os.environ.get('DLRM_ALLTOALL_IMPL', '')
+
+myreq = None
+
+def env2int(env_list, default = -1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0: return val
+    return default
+
+def get_my_slice(n):
+    my_size = dist.get_world_size()
+    my_rank = dist.get_rank()
+    k, m = divmod(n, my_size)
+    return slice(my_rank * k + min(my_rank, m), (my_rank+1) * k + min(my_rank+1, m), 1)
+
+def get_split_lengths(n):
+    my_size = dist.get_world_size()
+    k, m = divmod(n, my_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        my_rank = dist.get_rank()
+        splits = [(k+1) if i < m else k for i in range(my_size)]
+        my_len = splits[my_rank]
+    return (my_len, splits)
+
+def init_distributed(rank = -1, size = -1, backend=''):
+    global myreq
+    #global my_rank
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+    global allgatherv_supported
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(['PMI_SIZE', 'OMPI_COMM_WORLD_SIZE', 'MV2_COMM_WORLD_SIZE', 'WORLD_SIZE'])
+    if backend == '' and num_mpi_ranks > 1:
+        if oneccl_bindings_for_pytorch and env2int(['CCL_WORKER_COUNT']) > 0:
+            backend = 'ccl'
+        elif dist.is_mpi_available():
+            backend = 'mpi'
+        else:
+            print("WARNING: MPI multi-process launch detected but PyTorch MPI backend not available.")
+            backend = 'gloo'
+    if backend != '':
+        #guess Rank and size
+        if rank == -1:
+            rank = env2int(['PMI_RANK', 'OMPI_COMM_WORLD_RANK', 'MV2_COMM_WORLD_RANK', 'RANK'], 0)
+        if size == -1:
+            size = env2int(['PMI_SIZE', 'OMPI_COMM_WORLD_SIZE', 'MV2_COMM_WORLD_SIZE', 'WORLD_SIZE'], 1)
+        if not os.environ.get('RANK', None) and rank != -1: os.environ['RANK'] = str(rank)
+        if not os.environ.get('WORLD_SIZE', None) and size != -1: os.environ['WORLD_SIZE'] = str(size)
+        if not os.environ.get('MASTER_PORT', None): os.environ['MASTER_PORT'] = '29500'
+        if not os.environ.get('MASTER_ADDR', None):
+            local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+            if local_size != size and backend != 'mpi':
+                print("Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default")
+                print("If this run hangs, try exporting rank 0's hostname as MASTER_ADDR")
+            os.environ['MASTER_ADDR'] = '127.0.0.1'
+    if size > 1:
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        my_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if my_rank == 0: print("Running on %d ranks using %s backend" % (my_size, backend))
+        if backend == 'ccl':
+            print("Using CCL_ATL_TRANSPORT=%s" % os.environ.get('CCL_ATL_TRANSPORT', '(default)'))
+            print("Using CCL_ATL_SHM=%s" % os.environ.get('CCL_ATL_SHM', '(default)'))
+        if hasattr(dist, 'all_to_all_single'):
+            try:
+               # dist.all_to_all_single(torch.empty([0]), torch.empty([0]))
+                alltoall_supported = True
+            except RuntimeError:
+                pass
+        if a2a_impl == 'alltoall' and alltoall_supported == False:
+            print("Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall" % (a2a_impl, backend))
+            a2a_impl = 'scatter'
+        if a2a_impl != '': print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+        try:
+            x = torch.ones([my_rank])
+            y = torch.zeros([(my_size*(my_size-1))//2])
+            y = list(y.split([r for r in range(my_size)]))
+            dist.all_gather(y, x)
+            allgatherv_supported = True
+        except RuntimeError:
+            pass
+    else:
+        my_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    myreq = Request()
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2ai, *inputs):
+        global myreq
+        my_rank = dist.get_rank()
+        #print("All2All_ScatterList_Req:forward")
+        mb_split_lengths = a2ai.gNS if a2ai.gNS else a2ai.lN
+        emb_split_lengths = a2ai.gSS if a2ai.gSS else [a2ai.lS] * my_size
+        gather_list = []
+        req_list = []
+        for i in range(my_size):
+            for j in range(emb_split_lengths[i]):
+                out_tensor = inputs[0].new_empty([a2ai.lN, a2ai.E])
+                scatter_list = list(inputs[j].split(mb_split_lengths, dim = 0)) if i == my_rank else []
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2ai = a2ai
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        #print("All2All_ScatterList_Req:backward")
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        #print("All2All_Scatter_Wait:forward")
+        ctx.a2ai = myreq.a2ai
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        my_rank = dist.get_rank()
+        a2ai = ctx.a2ai
+        grad_output = [t.contiguous() for t in grad_output]
+        mb_split_lengths = a2ai.gNS if a2ai.gNS else [a2ai.lN] * my_size
+        per_rank_split_lengths = a2ai.gSS if a2ai.gSS else [a2ai.lS] * my_size
+        grad_inputs = [grad_output[0].new_empty([ctx.a2ai.N, ctx.a2ai.E]) for _ in range(a2ai.lS)]
+        req_list = []
+        ind = 0
+        for i in range(my_size):
+            for j in range(per_rank_split_lengths[i]):
+                gather_list = list(grad_inputs[j].split(mb_split_lengths, dim = 0)) if i == my_rank else None
+                req = dist.gather(grad_output[ind], gather_list, dst = i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2ai, *inputs):
+        global myreq
+        #print("All2All_Scatter_Req:forward")
+        my_rank = dist.get_rank()
+        mb_split_lengths = a2ai.gNS if a2ai.gNS else a2ai.lN
+        emb_split_lengths = a2ai.gSS if a2ai.gSS else [a2ai.lS] * my_size
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(mb_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(my_size):
+            out_tensor = input.new_empty([a2ai.lN, emb_split_lengths[i] * a2ai.E])
+            req = dist.scatter(out_tensor, scatter_list if i == my_rank else [], src=i, async_op=True)
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2ai = a2ai
+        ctx.a2ai = a2ai
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        #print("All2All_Scatter_Req:backward")
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2ai.E, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        #print("All2All_Scatter_Wait:forward")
+        ctx.a2ai = myreq.a2ai
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        my_rank = dist.get_rank()
+        #print("All2All_Scatter_Wait:backward")
+        assert len(grad_output) == my_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2ai = ctx.a2ai
+        mb_split_lengths = a2ai.gNS if a2ai.gNS else a2ai.lN
+        emb_split_lengths = a2ai.gSS if a2ai.gSS else [a2ai.lS] * my_size
+        grad_input = grad_output[0].new_empty([a2ai.N, a2ai.E*a2ai.lS])
+        gather_list = list(grad_input.split(mb_split_lengths, dim=0))
+        req_list = []
+        for i in range(my_size):
+            #req = dist.scatter(gather_list[i], scatter_list if i == my_rank else [], src=i, async_op=True)
+            req = dist.gather(scatter_list[i], gather_list if i == my_rank else [], dst=i, async_op=True)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2ai, *inputs):
+        global myreq
+        #print("All2All_Req:forward")
+        mb_split_lengths = a2ai.gNS
+        if mb_split_lengths: mb_split_lengths = [m * a2ai.lS * a2ai.E for m in mb_split_lengths]
+        emb_split_lengths = a2ai.gSS
+        if emb_split_lengths: emb_split_lengths = [a2ai.lN * e * a2ai.E for e in emb_split_lengths]
+        input = torch.cat(inputs, dim=1).view([-1])
+        output = input.new_empty([a2ai.S*a2ai.lN*a2ai.E])
+        req = dist.all_to_all_single(output, input, emb_split_lengths, mb_split_lengths, async_op=True)
+        myreq.req = req
+        myreq.tensor = []
+        myreq.tensor.append(output)
+        myreq.tensor = tuple(myreq.tensor)
+        a2ai.mb_split_lengths = mb_split_lengths
+        a2ai.emb_split_lengths = emb_split_lengths
+        myreq.a2ai = a2ai
+        ctx.a2ai = a2ai
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        #print("All2All_Req:backward")
+        a2ai = ctx.a2ai
+        myreq.req.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.view([a2ai.N, -1]).split(a2ai.E, dim=1)
+        grad_inputs = [gin.contiguous() for gin in grad_inputs]
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        #print("All2All_Wait:forward")
+        a2ai = myreq.a2ai
+        ctx.a2ai = a2ai
+        myreq.req.wait()
+        myreq.req = None
+        myreq.tensor = None
+        emb_split_lengths = a2ai.emb_split_lengths if a2ai.emb_split_lengths else a2ai.lS * a2ai.lN * a2ai.E
+        outputs = output[0].split(emb_split_lengths)
+        outputs = tuple([out.view([a2ai.lN, -1]) for out in outputs])
+        return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        #print("All2All_Wait:backward")
+        a2ai = ctx.a2ai
+        grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+        grad_output = torch.cat(grad_outputs)
+        grad_input = grad_output.new_empty([a2ai.N * a2ai.lS * a2ai.E])
+        req = dist.all_to_all_single(grad_input, grad_output, a2ai.mb_split_lengths, a2ai.emb_split_lengths, async_op=True)
+        myreq.req = req
+        myreq.tensor = grad_input
+        return (grad_output,)
+
+class AllGather(Function):
+
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * my_size
+        my_rank = dist.get_rank()
+        assert(len(global_lengths) == my_size)
+        assert(global_lengths[my_rank] == input.size(dim))
+        local_start = sum(global_lengths[:my_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[my_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(my_size)]
+            gather_list = []
+            for l in global_lengths:
+                output_size[dim] = l
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+class All2AllInfo(object):
+    pass
+
+def alltoall(inputs, per_rank_split_lengths):
+    global myreq
+    N, E = inputs[0].size()
+    a2ai = All2AllInfo()
+    a2ai.lS = len(inputs)
+    a2ai.gSS = per_rank_split_lengths
+    a2ai.lN, a2ai.gNS = get_split_lengths(N)
+    a2ai.E = E
+    a2ai.N = N
+    a2ai.S = sum(per_rank_split_lengths) if per_rank_split_lengths else a2ai.lS * my_size
+    if a2a_impl == '' and alltoall_supported or a2a_impl == 'alltoall':
+        output = All2All_Req.apply(a2ai, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == '' or a2a_impl == 'scatter':
+        #print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2ai, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == 'scatter_list':
+        #print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2ai, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print("Unknown value set for DLRM_ALLTOALL_IMPL (%s), please use one of [alltoall, scatter, scatter_list]" % a2a_impl) 
+    return myreq
+
+def shuffle_data(inputs):
+    input = torch.cat(inputs)
+    output = input.new_empty(input.size())
+    req = dist.all_to_all_single(output, input) 
+    output = output.reshape(my_size, -1)
+    return output
+    
+
+def all_gather(input, lengths, dim=0):
+    #print("lengths: ", lengths)
+    if not lengths: lengths = [input.size(0)] * my_size
+    return AllGather.apply(input, lengths, dim)
+
+def all_gather_validation(input, lengths, dim=0):
+    #print("lengths: ", lengths)
+    if not lengths: lengths = [input.size(0)] * my_size
+
+    global_lengths = lengths
+
+    if not isinstance(global_lengths, (list, tuple)):
+        global_lengths = [global_lengths] * my_size
+    my_rank = dist.get_rank()
+    assert(len(global_lengths) == my_size)
+    assert(global_lengths[my_rank] == input.size(dim))
+    local_start = sum(global_lengths[:my_rank])
+
+    output_size = list(input.size())
+
+    input = input.contiguous()
+    if dim == 0:
+        out_len = sum(global_lengths)
+        output_size[dim] = out_len
+        output = input.new_empty(output_size)
+        gather_list = list(output.split(global_lengths, dim=0))
+    else:
+        gather_list = [torch.empty_like(input) for _ in range(my_size)]
+        gather_list = []
+        for l in global_lengths:
+            output_size[dim] = l
+            gather_list.append(input.new_empty(output_size))
+
+    # if dim != 0:
+    #     output = torch.cat(gather_list, dim=dim)
+    req = dist.all_gather(gather_list, input, async_op=True)
+    return req, output
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/memrec.py b/models/recommendation/pytorch/memrec_dlrm/product/memrec.py
new file mode 100644
index 000000000..86b68d9f3
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/memrec.py
@@ -0,0 +1,1771 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Facebook, Inc. and its affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+# Description: an implementation of a deep learning recommendation model (DLRM)
+# The model input consists of dense and sparse features. The former is a vector
+# of floating point values. The latter is a list of sparse indices into
+# embedding tables, which consist of vectors of floating point values.
+# The selected vectors are passed to mlp networks denoted by triangles,
+# in some cases the vectors are interacted through operators (Ops).
+#
+# output:
+#                         vector of values
+# model:                        |
+#                              /\
+#                             /__\
+#                               |
+#       _____________________> Op  <___________________
+#     /                         |                      \
+#    /\                        /\                      /\
+#   /__\                      /__\           ...      /__\
+#    |                          |                       |
+#    |                         Op                      Op
+#    |                    ____/__\_____           ____/__\____
+#    |                   |_Emb_|____|__|    ...  |_Emb_|__|___|
+# input:
+# [ dense features ]     [sparse indices] , ..., [sparse indices]
+#
+# More precise definition of model layers:
+# 1) fully connected layers of an mlp
+# z = f(y)
+# y = Wx + b
+#
+# 2) embedding lookup (for a list of sparse indices p=[p1,...,pk])
+# z = Op(e1,...,ek)
+# obtain vectors e1=E[:,p1], ..., ek=E[:,pk]
+#
+# 3) Operator Op can be one of the following
+# Sum(e1,...,ek) = e1 + ... + ek
+# Dot(e1,...,ek) = [e1'e1, ..., e1'ek, ..., ek'e1, ..., ek'ek]
+# Cat(e1,...,ek) = [e1', ..., ek']'
+# where ' denotes transpose operation
+#
+# References:
+# [1] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang,
+# Narayanan Sundaram, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu,
+# Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii,
+# Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko,
+# Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong,
+# Misha Smelyanskiy, "Deep Learning Recommendation Model for Personalization and
+# Recommendation Systems", CoRR, arXiv:1906.00091, 2019
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+
+# miscellaneous
+import builtins
+import datetime
+import sys
+import time
+
+import random
+import hashlib
+import json
+import warnings
+import extend_distributed as ext_dist
+
+# data generation
+import memrec_data as dp
+
+# numpy
+import numpy as np
+import sklearn.metrics
+
+# pytorch
+import torch
+import torch.nn as nn
+from torch._ops import ops
+from torch.autograd.profiler import record_function
+from torch.nn.parallel.parallel_apply import parallel_apply
+from torch.nn.parallel.replicate import replicate
+from torch.nn.parallel.scatter_gather import gather, scatter
+from torch.nn.parameter import Parameter
+from torch.optim.lr_scheduler import _LRScheduler
+from torch.profiler import profile, record_function, ProfilerActivity
+
+from torch.utils.tensorboard import SummaryWriter
+# intel
+import intel_extension_for_pytorch as ipex
+from torch.utils import ThroughputBenchmark
+
+# int8
+from torch.ao.quantization import MinMaxObserver, PerChannelMinMaxObserver, QConfig
+from intel_extension_for_pytorch.quantization import prepare, convert
+# For distributed run
+import extend_distributed as ext_dist
+
+import os
+import psutil
+
+exc = getattr(builtins, "IOError", "FileNotFoundError")
+
+first_iteration_for_train = True
+
+buffer_num = 20
+data_buffer = None
+data_iter = None
+train_ld = None
+
+def freeze(model):
+    return torch.jit._recursive.wrap_cpp_module(torch._C._freeze_module(model._c, preserveParameters=True))
+
+
+def time_wrap():
+    return time.time()
+
+
+def dlrm_wrap(X, lS_o, lS_i, is_train=False):
+    with record_function("DLRM forward"):
+        return dlrm(X, lS_o, lS_i, is_train)
+
+
+def loss_fn_wrap(Z, T):
+    with record_function("DLRM loss compute"):
+        return dlrm.loss_fn(Z, T)
+
+# The following function is a wrapper to avoid checking this multiple times in th
+# loop below.
+def unpack_batch(b):
+    # Experiment with unweighted samples
+    return b[0], b[1], b[2], b[3], torch.ones(b[3].size()), None
+
+def load_data(buffer_num, is_bf16):
+    reset = False
+    with torch.autograd.profiler.record_function('load_data'):
+        for d in range(buffer_num):
+            # (X, lS_i, T) = next(data_iter)
+            try:
+                Batch = next(data_iter)
+            except:
+                print("epoch ended, reset data_iter")
+                reset = True
+                data_iter._reset(train_ld)
+                Batch = next(data_iter)
+            X, lS_o, lS_i, T, W, CBPP = unpack_batch(Batch)
+            X = X.bfloat16() if is_bf16 else X
+            data_buffer[d] = (X, lS_o, lS_i, T, W, CBPP)
+    return reset
+
+class AlltoallOutputs():
+    def __init__(self):
+        self.data = None
+        self.tail_data = None
+
+    def get_data_base_on_input(self, input, shape):
+        if self.data is None:
+            self.data = input.new_empty(shape)
+            return self.data
+        if self.data.shape == torch.Size(shape):
+            return self.data
+        else:
+            if self.tail_data is None:
+                self.tail_data = input.new_empty(shape)
+            return self.tail_data
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
+
+
+### define dlrm in PyTorch ###
+class DLRM_Net(nn.Module):
+    def create_mlp(self, ln, sigmoid_layer):
+        # build MLP layer by layer
+        layers = nn.ModuleList()
+        for i in range(0, ln.size - 1):
+            n = ln[i]
+            m = ln[i + 1]
+
+            # construct fully connected operator
+            LL = nn.Linear(int(n), int(m), bias=True)
+
+            # initialize the weights
+            # with torch.no_grad():
+            # custom Xavier input, output or two-sided fill
+            mean = 0.0  # std_dev = np.sqrt(variance)
+            std_dev = np.sqrt(2 / (m + n))  # np.sqrt(1 / m) # np.sqrt(1 / n)
+            W = np.random.normal(mean, std_dev, size=(m, n)).astype(np.float32)
+            std_dev = np.sqrt(1 / m)  # np.sqrt(2 / (m + 1))
+            bt = np.random.normal(mean, std_dev, size=m).astype(np.float32)
+            # approach 1
+            LL.weight.data = torch.tensor(W, requires_grad=True)
+            LL.bias.data = torch.tensor(bt, requires_grad=True)
+            # approach 2
+            # LL.weight.data.copy_(torch.tensor(W))
+            # LL.bias.data.copy_(torch.tensor(bt))
+            # approach 3
+            # LL.weight = Parameter(torch.tensor(W),requires_grad=True)
+            # LL.bias = Parameter(torch.tensor(bt),requires_grad=True)
+            layers.append(LL)
+            self.numel += (LL.weight.numel() + LL.bias.numel())
+
+            # construct sigmoid or relu operator
+            if i == sigmoid_layer:
+                layers.append(nn.Sigmoid())
+            else:
+                layers.append(nn.ReLU())
+
+        # approach 1: use ModuleList
+        # return layers
+        # approach 2: use Sequential container to wrap all layers
+        return torch.nn.Sequential(*layers)
+
+    def create_emb(self, m, ln, local_ln_emb_sparse, ln_emb_dense, np_init_emb_weight=False):
+        total_numel = 0
+        for n in ln:
+            total_numel += m * n
+        print("start to create emb, minimally need ~{} G memory ".format(total_numel * 4 / 1024 / 1024 / 1024))
+        print("may actually use more memory depends on memory allocator like tcmalloc/jemalloc")
+        print("current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        emb_l = nn.ModuleList()
+        emb_dense = nn.ModuleList()
+        emb_sparse = nn.ModuleList()
+        #n_embs = ln.size if local_ln_emb is None else len(local_ln_emb)
+        embs = range(len(ln))
+        if local_ln_emb_sparse or ln_emb_dense:
+            embs = local_ln_emb_sparse + ln_emb_dense
+        for i in embs:
+            n = ln[i]
+            sparse_grad = n >= self.sparse_dense_boundary and ext_dist.my_size > 1
+            if np_init_emb_weight:
+                W = np.random.uniform(
+                        low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, m)
+                    ).astype(np.float32)
+                # W = np.zeros(shape=(n, m)).astype(np.float32)
+                EE = nn.EmbeddingBag(n, m, mode="sum", sparse=sparse_grad, _weight=torch.from_numpy(W).requires_grad_())
+            else:
+                EE = nn.EmbeddingBag(n, m, mode="sum", sparse=sparse_grad)
+            if ext_dist.my_size > 1 and n >= self.sparse_dense_boundary:
+                emb_sparse.append(EE)
+            elif ext_dist.my_size > 1:
+                emb_dense.append(EE)
+            else:
+                emb_l.append(EE)
+        print("create emb done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        self.numel += total_numel
+        return emb_l, emb_dense, emb_sparse
+    
+    def create_emb_bloomrec(self):
+        n = D
+        m = np.fromstring(args.arch_mlp_bot_sparse, dtype=int, sep="-")[0]
+        
+        EE = nn.EmbeddingBag(n, m, mode="sum", sparse=True)
+        W = np.random.uniform(low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, m)).astype(np.float32)
+        EE.weight.data = torch.tensor(W, requires_grad=True)
+
+        n_dash = dw
+        m_dash = 1
+        EE_dash = nn.EmbeddingBag(n_dash, m_dash, mode="sum", sparse=True)
+        W_dash = np.random.uniform(low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n_dash, m_dash)).astype(np.float32)
+        EE_dash.weight.data = torch.tensor(W_dash, requires_grad=True)
+
+
+        return EE, EE_dash
+    
+    def __init__(
+        self,
+        m_spa=None,
+        ln_emb=None,
+        ln_bot=None,
+        ln_bot_sparse=None,
+        ln_top=None,
+        sigmoid_bot=-1,
+        sigmoid_top=-1,
+        weighted_pooling=None,
+        loss_threshold=0.0,
+        np_init_emb_weight=False,
+        sparse_dense_boundary = 2048,
+        K=None,
+        kw=None
+    ):
+        print("np_init_emb_weight:" , np_init_emb_weight)
+        super(DLRM_Net, self).__init__()
+        self.numel = 0
+        self.loss_threshold = loss_threshold
+
+        n_emb = len(ln_emb)
+        self.sparse_dense_boundary = sparse_dense_boundary
+       
+        self.ln_emb_dense = [i for i in range(n_emb) if ln_emb[i] < self.sparse_dense_boundary]
+        self.ln_emb_sparse = [i for i in range(n_emb) if ln_emb[i] >= self.sparse_dense_boundary]
+        
+        #If running distributed, get local slice of embedding tables
+        self.rank = -1
+        if ext_dist.my_size > 1:
+            self.rank = ext_dist.dist.get_rank()
+            n_emb_sparse = len(self.ln_emb_sparse)
+            self.n_local_emb_sparse, self.n_sparse_emb_per_rank = ext_dist.get_split_lengths(n_emb_sparse)
+            self.local_ln_emb_sparse_slice = ext_dist.get_my_slice(n_emb_sparse)
+            self.local_ln_emb_sparse = self.ln_emb_sparse[self.local_ln_emb_sparse_slice]
+        self.l_emb_seeds = np.random.randint(low=0, high=100000, size=len(ln_emb))
+        print("#############seed self.l_emb_seeds:", self.l_emb_seeds)
+        self.bot_l = self.create_mlp(ln_bot, sigmoid_bot)
+        self.top_l = self.create_mlp(ln_top, sigmoid_top)
+        self.all2all_validation_outputs0 = AlltoallOutputs()
+        self.all2all_validation_outputs1 = AlltoallOutputs()
+        self.output_ind = AlltoallOutputs()
+        self.validation_output_ind0 = AlltoallOutputs()
+        self.validation_output_ind1 = AlltoallOutputs()
+        if ext_dist.my_size > 1:
+            self.emb_l, self.emb_dense, self.emb_sparse = self.create_emb(m_spa, ln_emb, self.local_ln_emb_sparse, self.ln_emb_dense,np_init_emb_weight)
+        else:
+            self.emb_l, self.emb_w = self.create_emb_bloomrec()
+        self.loss_fn = torch.nn.BCELoss(reduction="mean")
+        self.emb_args_preprocessing = 0
+        self.K=K
+        self.kw=kw
+                
+    def apply_mlp(self, x, layers):
+        # approach 1: use ModuleList
+         # for layer in layers:
+        #     x = layer(x)
+        # return x
+        # approach 2: use Sequential container to wrap all layers
+        return layers(x)
+
+    def apply_emb(self, emb_l, lS_o, lS_i):
+        # WARNING: notice that we are processing the batch at once. We implicitly
+        # assume that the data is laid out such that:
+        # 1. each embedding is indexed with a group of sparse indices,
+        #   corresponding to a single lookup
+        # 2. for each embedding the lookups are further organized into a batch
+        # 3. for a list of embedding tables there is a list of batched lookups
+        is_ddp = isinstance(emb_l,  ext_dist.DDP)
+        
+        if (not is_ddp and isinstance(emb_l,ipex.nn.modules.MergedEmbeddingBag)) or (is_ddp and isinstance(emb_l.module,ipex.nn.modules.MergedEmbeddingBag)): 
+             start = time.time()
+             n_tables = len(lS_i)
+             #prepare the input for sparse mergedembedding
+             idx = [lS_i[i] for i in range(n_tables)]
+             offset = [lS_o[i] for i in range(n_tables)]
+             include_last = [False for i in range(n_tables)]
+             if is_ddp:
+                 indices, offsets, indices_with_row_offsets = emb_l.module.linearize_indices_and_offsets(idx, offset, include_last)
+             else:
+                 indices, offsets, indices_with_row_offsets = emb_l.linearize_indices_and_offsets(idx, offset, include_last)
+             emb_args = (indices, offsets, indices_with_row_offsets)
+             self.emb_args_preprocessing += time.time() - start
+             return emb_l(emb_args, self.need_linearize_indices_and_offsets)
+        ly = []
+        for k, sparse_index_group_batch in enumerate(lS_i):
+            sparse_offset_group_batch = lS_o[k]
+
+            # embedding lookup
+            # We are using EmbeddingBag, which implicitly uses sum operator.
+            # The embeddings are represented as tall matrices, with sum
+            # happening vertically across 0 axis, resulting in a row vector
+            E = emb_l[k]
+            V = E(
+                sparse_index_group_batch,
+                sparse_offset_group_batch,
+            )
+
+            ly.append(V)
+
+        return ly
+
+    def apply_emb_bloomrec(self, lS_o, lS_i, emb_l, emb_w):
+        #lS_i : batch_size*26*(K+kw) : First K fraction entries is for 1st etable, remaining for the 2nd etable
+        #lS_0 : two equal halves of batch size, exact values of each halves depends on K and kw
+        first_part_i = int(int(lS_i.size()[0])*self.K/(self.K+self.kw))
+        first_part_o = int(int(lS_o.size()[0])/2)
+
+        E = emb_l
+        lS_i_ = lS_i[0:first_part_i]
+        lS_o_ = lS_o[0:first_part_o]
+        V = E(lS_i_.type(torch.LongTensor), lS_o_.type(torch.LongTensor))
+
+        E_w = emb_w
+        lS_i_w = lS_i[first_part_i:]
+        lS_o_w = lS_o[first_part_o:]
+        V_w = E_w(lS_i_w.type(torch.LongTensor), lS_o_w.type(torch.LongTensor))
+        
+        if (self.kw == 0):
+            return (V)
+        else:
+            return (V*V_w)
+    
+    def interact_features(self, x, ly):
+        if args.ipex_interaction:
+            T = [x] + list(ly)
+            R = ipex.nn.functional.interaction(*T)
+        else:
+            # concatenate dense and sparse features
+            R = torch.cat([x] + [ly], dim=1)
+        return R
+
+    def forward(self, dense_x, lS_o, lS_i, is_train=False):
+        if ext_dist.my_size > 1:
+            return self.distributed_forward(dense_x, lS_o, lS_i, is_train=is_train)
+        else:
+            return self.sequential_forward(dense_x, lS_o, lS_i)
+
+    def distributed_forward_2iter_overlap(self, cur_data, next_data):
+        if not cur_data[0]:
+            return None, None
+
+        dense_x0 = cur_data[0]['dense_x']
+        T_test0 = cur_data[0]['T_test']
+        lS_i0 = cur_data[0]['lS_i']
+        lS_o0 = cur_data[0]['lS_o']
+        batch_size0 = dense_x0.size()[0]
+
+        lS_i_dense0 = [lS_i0[i] for i in self.ln_emb_dense]
+        lS_i_sparse0 = [lS_i0[i] for i in self.ln_emb_sparse]
+
+        if not cur_data[1]:
+            lS_i_sparse0 = ext_dist.shuffle_data(lS_i_sparse0)
+            g_i_sparse0 = [lS_i_sparse0[:, i * batch_size0:(i + 1) * batch_size0].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+            offset = torch.arange(batch_size0 * ext_dist.my_size)
+            g_o_sparse0 = [offset for i in range(self.n_local_emb_sparse)]
+
+            # sparse embeddings
+            with torch.autograd.profiler.record_function('Prof_sparse_emb_forward'):
+                ly_sparse0 = self.apply_emb(dlrm.emb_sparse, g_o_sparse0, g_i_sparse0)
+
+            with torch.autograd.profiler.record_function('Prof_alltoall_emb_forward'):
+                a2a_req0 = ext_dist.alltoall(ly_sparse0, self.n_sparse_emb_per_rank)
+
+            # dense embeddings
+            with torch.autograd.profiler.record_function('Prof_dense_emb_forward'):
+                ly_dense0 = self.apply_emb(batch_size0, lS_i_dense0, self.emb_dense)
+
+            # bottom mlp
+            # with torch.autograd.profiler.record_function('Prof_bot_mlp_forward'):
+            x0 = self.apply_mlp(dense_x0, self.bot_l)
+            with torch.autograd.profiler.record_function('Prof_alltoall_emb_wait_forward'):
+                ly_sparse0 = a2a_req0.wait()
+
+            # concat emb data for split sparse embs
+            ly_sparse0_full = []
+            if ext_dist.my_size > len(self.ln_emb_sparse):
+                for i in range(len(self.ln_emb_sparse)):
+                    ly_sparse0_split = torch.cat([ly_sparse0[j] for j in range(i, ext_dist.my_size, len(self.ln_emb_sparse))], 1)
+                    ly_sparse0_full.append(ly_sparse0_split)
+            else:
+                ly_sparse0_full = list(ly_sparse0)
+
+            ly0 = list(ly_dense0) + ly_sparse0_full
+
+            with torch.autograd.profiler.record_function('Prof_interaction_forward'):
+                z0 = self.interact_features(x0, ly0)
+            # top mlp
+            with torch.autograd.profiler.record_function('Prof_top_mlp_forward'):
+                p0 = self.apply_mlp(z0, self.top_l)
+
+            # clamp output if needed
+            if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+                z0 = torch.clamp(
+                    p0, min=self.loss_threshold, max=(1.0 - self.loss_threshold)
+                )
+            else:
+                z0 = p0
+            z0 = z0.float()
+
+            Z_test0 = ext_dist.all_gather(z0, None)
+            T_test0 = ext_dist.all_gather(T_test0, None)
+
+            return (Z_test0, T_test0), None
+
+        # if not cur_data[1]:
+        #     self.distributed_forward(cur_data[0])
+
+        if cur_data[1]:
+            dense_x1 = cur_data[1]['dense_x']
+            T_test1 = cur_data[1]['T_test']
+            lS_i1 = cur_data[1]['lS_i']
+            lS_o1 = cur_data[1]['lS_o']
+            batch_size1 = dense_x1.size()[0]
+
+            lS_i_dense1 = [lS_i1[i] for i in self.ln_emb_dense]
+            lS_i_sparse1 = [lS_i1[i] for i in self.ln_emb_sparse]
+
+        if next_data[0]:
+            next_dense_x0 = next_data[0]['dense_x']
+            # next_T_test0 = next_data[0]['T_test']
+            next_lS_i0 = next_data[0]['lS_i']
+            next_lS_o0 = next_data[0]['lS_o']
+            next_batch_size0 = next_dense_x0.size()[0]
+
+            # next_lS_i_dense0 = [lS_i0[i] for i in self.ln_emb_dense]
+            next_lS_i_sparse0 = [next_lS_i0[i] for i in self.ln_emb_sparse]
+
+        if next_data[1]:
+            next_dense_x1 = next_data[1]['dense_x']
+            # next_T_test1 = next_data[1]['T_test']
+            next_lS_i1 = next_data[1]['lS_i']
+            next_lS_o1 = next_data[1]['lS_o']
+            next_batch_size1 = next_dense_x1.size()[0]
+
+            # next_lS_i_dense1 = [lS_i1[i] for i in self.ln_emb_dense]
+            next_lS_i_sparse1 = [next_lS_i1[i] for i in self.ln_emb_sparse]
+
+        if 'lS_i_sparse_res' not in cur_data[0]:
+            if ext_dist.my_size > len(self.ln_emb_sparse):
+                num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
+                lS_i_sparse0 = torch.cat([lS_i_sparse0 for _ in range(num_split_grps) ])
+            lS_i_sparse0 = ext_dist.shuffle_data(lS_i_sparse0)
+            g_i_sparse0 = [lS_i_sparse0[:, i * batch_size0:(i + 1) * batch_size0].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+        else:
+            g_i_sparse0 = cur_data[0]['lS_i_sparse_res']
+
+        if cur_data[1] and 'lS_i_sparse_res' not in cur_data[1]:
+            lS_i_sparse1 = torch.cat(lS_i_sparse1)
+            output1 = lS_i_sparse1.new_empty(lS_i_sparse1.size())
+            ind_req1 = ext_dist.dist.all_to_all_single(output1, lS_i_sparse1, async_op=True)
+
+        if 'ly_sparse' not in cur_data[0]:
+            with torch.autograd.profiler.record_function('Prof_sparse_emb_forward'):
+                # sparse0 embeddings
+                offset = torch.arange(batch_size0 * ext_dist.my_size)
+                g_o_sparse0 = [offset for i in range(self.n_local_emb_sparse)]
+                ly_sparse0 = self.apply_emb(dlrm.emb_sparse, g_o_sparse0, g_i_sparse0)
+        else:
+            ly_sparse0 = cur_data[0]['ly_sparse']
+
+        if cur_data[1] and 'lS_i_sparse_res' not in cur_data[1]:
+            ind_req1.wait()
+            lS_i_sparse1 = output1.reshape(ext_dist.my_size, -1)
+            g_i_sparse1 = [lS_i_sparse1[:, i * batch_size1:(i + 1) * batch_size1].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+        else:
+            g_i_sparse1 = cur_data[1]['lS_i_sparse_res']
+
+
+        with torch.autograd.profiler.record_function('Prof_alltoall_emb_forward'):
+            a2a_req0 = ext_dist.alltoall(ly_sparse0, self.n_sparse_emb_per_rank)
+
+        # dense0 embeddings
+        with torch.autograd.profiler.record_function('Prof_dense_emb_forward'):
+            ly_dense0 =  self.apply_emb(dlrm.emb_dense, lS_o0, lS_i_dense0)
+
+        # bottom mlp 0
+        with torch.autograd.profiler.record_function('Prof_bot_mlp_forward'):
+            x0 = self.apply_mlp(dense_x0, self.bot_l)
+        
+        if cur_data[1]:
+            # sparse1 embeddings
+            with torch.autograd.profiler.record_function('Prof_sparse_emb_forward1'):
+                offset = torch.arange(batch_size1 * ext_dist.my_size)
+                g_o_sparse1 = [offset for i in range(self.n_local_emb_sparse)]
+                ly_sparse1 = self.apply_emb(dlrm.emb_sparse, g_o_sparse1, g_i_sparse1)
+
+        with torch.autograd.profiler.record_function('Prof_alltoall_emb_wait_forward'):
+            ly_sparse0 = a2a_req0.wait()
+
+        T_test_req0, T_test0 = ext_dist.all_gather_validation(T_test0, None)
+
+        # concat emb data for split sparse0 embs
+        ly_sparse0_full = []
+        if ext_dist.my_size > len(self.ln_emb_sparse):
+            for i in range(len(self.ln_emb_sparse)):
+                ly_sparse0_split = torch.cat([ly_sparse0[j] for j in range(i, ext_dist.my_size, len(self.ln_emb_sparse))], 1)
+                ly_sparse0_full.append(ly_sparse0_split)
+        else:
+            vector_lenght = 128
+            for item in ly_sparse0:
+                ly_sparse0_full += [item[:, emb_id * vector_lenght: (emb_id + 1) * vector_lenght] for emb_id in range(self.emb_sparse.n_tables)]
+            ly_sparse0_full = [item.contiguous() for item in ly_sparse0_full]
+            ly_sparse0_full = list(ly_sparse0_full)
+
+        ly0 = list(ly_dense0) + ly_sparse0_full
+
+        with torch.autograd.profiler.record_function('Prof_interaction_forward'):
+            z0 = self.interact_features(x0, ly0)
+
+        with torch.autograd.profiler.record_function('Prof_all_gather_t0'):
+            T_test_req0.wait()
+
+        if cur_data[1]:
+            with torch.autograd.profiler.record_function('Prof_alltoall_emb_forward1'):
+                a2a_req1 = ext_dist.alltoall(ly_sparse1, self.n_sparse_emb_per_rank)
+
+        if next_data[0]:
+            if ext_dist.my_size > len(self.ln_emb_sparse):
+                num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
+                next_lS_i_sparse0 = torch.cat([next_lS_i_sparse0 for _ in range(num_split_grps) ])
+            next_lS_i_sparse0 = torch.cat(next_lS_i_sparse0)
+            next_output0 = self.validation_output_ind0.get_data_base_on_input(next_lS_i_sparse0, next_lS_i_sparse0.size())
+            next_ind_req0 = ext_dist.dist.all_to_all_single(next_output0, next_lS_i_sparse0, async_op=True)
+
+        # top mlp 0
+        with torch.autograd.profiler.record_function('Prof_top_mlp_forward'):
+            p0 = self.apply_mlp(z0, self.top_l)
+
+        if cur_data[1]:
+            with torch.autograd.profiler.record_function('Prof_alltoall_emb_wait_forward1'):
+                ly_sparse1 = a2a_req1.wait()
+
+        if next_data[0]:
+            next_ind_req0.wait()
+            next_lS_i_sparse0 = next_output0.reshape(ext_dist.my_size, -1)
+            next_g_i_sparse0 = [next_lS_i_sparse0[:, i * next_batch_size0:(i + 1) * next_batch_size0].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+            next_data[0]['lS_i_sparse_res'] = next_g_i_sparse0
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z0 = torch.clamp(
+                p0, min=self.loss_threshold, max=(1.0 - self.loss_threshold)
+            )
+        else:
+            z0 = p0
+        z0 = z0.float()
+
+        Z_test_req0, Z_test0 = ext_dist.all_gather_validation(z0, None)
+
+        if cur_data[1]:
+            # dense1 embeddings
+            with torch.autograd.profiler.record_function('Prof_dense_emb_forward1'):
+                ly_dense1 =  self.apply_emb(dlrm.emb_dense, lS_o1, lS_i_dense1)
+            # bottom mlp 1
+            with torch.autograd.profiler.record_function('Prof_bot_mlp_forward1'):
+                x1 = self.apply_mlp(dense_x1, self.bot_l)
+
+        with torch.autograd.profiler.record_function('Prof_all_gather_z0'):
+            Z_test_req0.wait()
+
+        if cur_data[1]:
+            T_test_req1, T_test1 = ext_dist.all_gather_validation(T_test1, None)
+
+            # concat emb data for split sparse1 embs
+            ly_sparse1_full = []
+            vector_lenght = 128
+            for item in ly_sparse1:
+                ly_sparse1_full += [item[:, emb_id * vector_lenght: (emb_id + 1) * vector_lenght] for emb_id in range(self.emb_sparse.n_tables)]
+            ly_sparse1_full = [item.contiguous() for item in ly_sparse1_full]
+            ly_sparse1_full = list(ly_sparse1_full)
+
+            ly1 = list(ly_dense1) + ly_sparse1_full
+
+            with torch.autograd.profiler.record_function('Prof_interaction_forward1'):
+                z1 = self.interact_features(x1, ly1)
+
+        if next_data[1]:
+            if ext_dist.my_size > len(self.ln_emb_sparse):
+                num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
+                next_lS_i_sparse1 = torch.cat([next_lS_i_sparse1 for _ in range(num_split_grps) ])
+            next_lS_i_sparse1 = torch.cat(next_lS_i_sparse1)
+            next_output1 = self.validation_output_ind1.get_data_base_on_input(next_lS_i_sparse1, next_lS_i_sparse1.size())
+            next_ind_req1 = ext_dist.dist.all_to_all_single(next_output1, next_lS_i_sparse1, async_op=True)
+
+        if cur_data[1]:
+            p1 = self.apply_mlp(z1, self.top_l)
+
+            with torch.autograd.profiler.record_function('Prof_all_gather_t1'):
+                T_test_req1.wait()
+
+            # clamp output if needed
+            if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+                z1 = torch.clamp(
+                    p1, min=self.loss_threshold, max=(1.0 - self.loss_threshold)
+                )
+            else:
+                z1 = p1
+            z1 = z1.float()
+
+        if next_data[1]:
+            next_ind_req1.wait()
+            next_lS_i_sparse1 = next_output1.reshape(ext_dist.my_size, -1)
+            next_g_i_sparse1 = [next_lS_i_sparse1[:, i * next_batch_size1:(i + 1) * next_batch_size1].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+            next_data[1]['lS_i_sparse_res'] = next_g_i_sparse1
+
+        if cur_data[1]:
+            Z_test_req1, Z_test1 = ext_dist.all_gather_validation(z1, None)
+
+        if next_data[0]:
+            offset = torch.arange(next_batch_size0 * ext_dist.my_size)
+            next_g_o_sparse0 = [offset for i in range(self.n_local_emb_sparse)]
+            next_ly_sparse0 = self.apply_emb(dlrm.emb_sparse, next_g_o_sparse0, next_data[0]['lS_i_sparse_res'])
+            next_data[0]['ly_sparse'] = next_ly_sparse0
+
+        if cur_data[1]:
+            Z_test_req1.wait()
+
+        if cur_data[1]:
+            return (Z_test0, T_test0), (Z_test1, T_test1)
+        else:
+            return (Z_test0, T_test0), None
+
+
+    def distributed_forward(self, dense_x, lS_o, lS_i, is_train=False):
+        local_batch_size = dense_x.size()[0]
+
+        #vector_lenght = self.emb_l.weights[0].size()[1]
+        # WARNING: # of ranks must be <= batch size in distributed_forward call
+        if local_batch_size < ext_dist.my_size:
+            sys.exit("ERROR: local_batch_size (%d) must be larger than number of ranks (%d)" % (local_batch_size, ext_dist.my_size))
+        
+        rank_id = dlrm.rank
+        #print("#####################rank_id:", rank_id)
+        if not isinstance(lS_i, list):
+            lS_i_dense = [lS_i[i] for i in self.ln_emb_dense]
+            lS_i_sparse = [lS_i[i] for i in self.ln_emb_sparse]
+
+        lS_o_dense = [lS_o[i] for i in self.ln_emb_dense]
+        lS_o_sparse = [lS_o[i] for i in self.ln_emb_sparse]
+        # shuffle_start = time.time()
+        # #lS_i_sparse = ext_dist.shuffle_data(lS_i_sparse)
+        # lS_i_sparse = torch.cat(lS_i_sparse)
+        # output_lS_i_sparse = lS_i_sparse.new_empty(lS_i_sparse.size())
+        # #print("cat time:", 1000*(time.time() - shuffle_start))
+        # req = ext_dist.dist.all_to_all_single(output_lS_i_sparse, lS_i_sparse)
+        # lS_i_sparse = output_lS_i_sparse.reshape(ext_dist.my_size, -1)
+
+        # g_i_sparse = [lS_i_sparse[:, i * local_batch_size:(i + 1) * local_batch_size].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
+
+        global first_iteration_for_train
+        if first_iteration_for_train or not is_train: 
+            #lS_i_sparse = ext_dist.shuffle_data(lS_i_sparse)
+            lS_i_sparse = torch.cat(lS_i_sparse)
+            output_lS_i_sparse = lS_i_sparse.new_empty(lS_i_sparse.size())
+            #print("cat time:", 1000*(time.time() - shuffle_start))
+            req = ext_dist.dist.all_to_all_single(output_lS_i_sparse, lS_i_sparse)
+            lS_i_sparse = output_lS_i_sparse.reshape(ext_dist.my_size, -1)
+            g_i_sparse = [lS_i_sparse[:, i * local_batch_size:(i + 1) * local_batch_size].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
+            if is_train:
+                first_iteration_for_train = False
+        else:
+            g_i_sparse = lS_i[1]
+            lS_i_dense = lS_i[0]
+
+        offset = torch.arange(local_batch_size * ext_dist.my_size)
+        #print("shuffle time:", 1000*(shuffle_end - shuffle_start))
+
+        g_o_sparse = [offset for i in range(self.n_local_emb_sparse)]
+        ly_sparse = self.apply_emb(dlrm.emb_sparse, g_o_sparse, g_i_sparse)
+        a2a_req = ext_dist.alltoall(ly_sparse, self.n_sparse_emb_per_rank)
+
+        if is_train:
+            load_data(buffer_num, args.bf16)
+
+        #dense embedding 
+        ly_dense =  self.apply_emb(dlrm.emb_dense, lS_o_dense, lS_i_dense)
+        # bottom mlp
+        x = self.apply_mlp(dense_x, self.bot_l)
+        ly_sparse = a2a_req.wait()
+        _ly = []
+        vector_lenght = 128
+        for item in ly_sparse:
+            _ly += [item[:, emb_id * vector_lenght: (emb_id + 1) * vector_lenght] for emb_id in range(self.emb_sparse.n_tables)]
+        _ly = [item.contiguous() for item in _ly]
+        _ly = list(ly_dense) + list(_ly)
+
+        a2a_ind_req = None #ovlerlap the a2a_ind_req with interaction/top_mlp
+        if is_train:
+            #get global index for sparse embedding
+            (X_next, lS_o_next, lS_i_next, T_next, W_next, CBPP_next) = data_buffer[0]
+            lS_i_dense_next = [lS_i_next[i] for i in self.ln_emb_dense]
+            lS_i_sparse_next = [lS_i_next[i] for i in self.ln_emb_sparse]
+            if ext_dist.my_size > len(self.ln_emb_sparse):
+                num_split_grps = ext_dist.my_size // len(self.ln_emb_sparse)
+                lS_i_sparse_next = torch.cat([lS_i_sparse_next for _ in range(num_split_grps) ])
+            lS_i_sparse_next = torch.cat(lS_i_sparse_next)
+            output_lS_i_sparse_next = lS_i_sparse_next.new_empty(lS_i_sparse_next.size())
+            # output_ind = self.output_ind.get_data_base_on_input(lS_i_sparse_next, lS_i_sparse_next.size())
+            a2a_ind_req = ext_dist.dist.all_to_all_single(output_lS_i_sparse_next, lS_i_sparse_next, async_op=True)
+
+        # interactions
+        z = self.interact_features(x, _ly)
+        # top mlp
+        p = self.apply_mlp(z, self.top_l)
+
+        if is_train:
+            a2a_ind_req.wait()
+            lS_i_sparse_next = output_lS_i_sparse_next.reshape(ext_dist.my_size, -1)
+            batch_size_next = X_next.size()[0]
+            # g_i_sparse = [lS_i_sparse_next[:, i * batch_size_next:(i + 1) * batch_size_next].reshape(-1).contiguous() for i in range(len(self.local_ln_emb_sparse))]
+            g_i_sparse_next = [lS_i_sparse_next[:, i * batch_size_next:(i + 1) * batch_size_next].reshape(-1) for i in range(len(self.local_ln_emb_sparse))]
+            # lS_i_sparse_next = torch.cat(g_i_sparse)
+            # import pdb
+            # pdb.set_trace()
+            # for i in self.ln_emb_sparse:
+            #    lS_i_next[i] = lS_i_sparse_next[i]
+            data_buffer[0] = (X_next, lS_o_next, [lS_i_dense_next, g_i_sparse_next], T_next, W_next, CBPP_next)
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z = torch.clamp(
+                p, min=self.loss_threshold, max=(1.0 - self.loss_threshold)
+            )                                                                         
+        else:
+            z = p
+        return z
+ 
+
+    def sequential_forward(self, dense_x, lS_o, lS_i):
+        # process dense features (using bottom mlp), resulting in a row vector
+        self.emb_args_preprocessing = 0
+
+        x = self.apply_mlp(dense_x, self.bot_l)
+
+        # process sparse features(using embeddings), resulting in a list of row vectors
+        ly = self.apply_emb_bloomrec(lS_o, lS_i, self.emb_l, self.emb_w)
+
+        # interact features (dense and sparse)
+        z = self.interact_features(x, ly)
+
+        # obtain probability of a click (using top mlp)
+        p = self.apply_mlp(z, self.top_l)
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z = torch.clamp(p, min=self.loss_threshold, max=(1.0 - self.loss_threshold))
+        else:
+            z = p
+       
+        return z
+
+
+def dash_separated_ints(value):
+    vals = value.split("-")
+    for val in vals:
+        try:
+            int(val)
+        except ValueError:
+            raise argparse.ArgumentTypeError(
+                "%s is not a valid dash separated list of ints" % value
+            )
+
+    return value
+
+
+def trace_model(args, dlrm, test_ld):
+    dlrm.eval()
+    for j, inputBatch in enumerate(test_ld):
+        X, lS_o, lS_i, _, _, _ = unpack_batch(inputBatch)
+        if args.bf32:
+            ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+        if args.bf16:
+            # at::GradMode::is_enabled() will query a threadlocal flag
+            # but new thread generate from throughputbench mark will 
+            # init this flag to true, so we temporal cast embedding's
+            # weight to bfloat16 for now
+            if args.inference_only:
+                dlrm.emb_l.bfloat16()
+            dlrm = ipex.optimize(dlrm, dtype=torch.bfloat16, inplace=True)
+        elif args.int8:
+            if args.num_cpu_cores != 0:
+                torch.set_num_threads(args.num_cpu_cores)
+            qconfig = QConfig(activation=MinMaxObserver.with_args(qscheme=torch.per_tensor_symmetric, dtype=torch.qint8),
+                weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric))
+            prepare(dlrm, qconfig, example_inputs=(X, lS_o, lS_i), inplace=True)
+            dlrm.load_qconf_summary(qconf_summary = args.int8_configure)
+            convert(dlrm, inplace=True)
+        else:
+            if args.bf32:
+                ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+            dlrm = ipex.optimize(dlrm, dtype=torch.float, inplace=True, auto_kernel_selection=True)
+        if args.int8:
+            print("Start to trace/freeze for int8, may need {} to save int8 weight".format(dlrm.numel / 1024 / 1024 / 1024))
+            print("Current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+            dlrm = torch.jit.trace(dlrm, [X, lS_o, lS_i])
+            dlrm = torch.jit.freeze(dlrm)
+            print("After trace/freeze, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+            dlrm(X, lS_o, lS_i)
+            dlrm(X, lS_o, lS_i)
+        else:
+            with torch.cpu.amp.autocast(enabled=args.bf16):
+                dlrm = torch.jit.trace(dlrm, (X, lS_o, lS_i), check_trace=True)
+                dlrm = torch.jit.freeze(dlrm)
+        dlrm(X, lS_o, lS_i)
+        dlrm(X, lS_o, lS_i)
+        return dlrm
+
+
+def run_throughput_benchmark(args, dlrm, test_ld):
+    if args.num_cpu_cores != 0:
+        torch.set_num_threads(1)
+    bench = ThroughputBenchmark(dlrm)
+    for j, inputBatch in enumerate(test_ld):
+        X, lS_o, lS_i, T, W, CBPP = unpack_batch(inputBatch)
+        bench.add_input(X, lS_o, lS_i)
+        if j == 1000: 
+            break
+    stats = bench.benchmark(
+        num_calling_threads=args.share_weight_instance,
+        num_warmup_iters=100,
+        num_iters=args.num_batches * args.share_weight_instance,
+    )
+    print(stats)
+    latency = stats.latency_avg_ms
+    throughput = (1 / latency) * 1000 * args.test_mini_batch_size * args.share_weight_instance
+    print("Throughput: {:.3f} fps".format(throughput))
+    sys.exit()
+
+
+def inference(
+    args,
+    dlrm,
+    best_acc_test,
+    best_auc_test,
+    test_ld,
+):
+    test_accu = 0
+    test_samp = 0
+
+    if args.print_auc:
+        scores = []
+        targets = []
+
+    total_time = 0
+    total_iter = 0
+    if args.inference_only:
+        dlrm = trace_model(args, dlrm, test_ld)
+    if args.share_weight_instance != 0:
+        run_throughput_benchmark(args, dlrm, test_ld)
+    with torch.cpu.amp.autocast(enabled=args.bf16):
+        if ext_dist.my_size > 1:
+            i = 0
+            n_test_iter = 2     # exec 2 iter per validation
+            cur_data = [dict() for _ in range(n_test_iter)]
+            next_data = [dict() for _ in range(n_test_iter)]
+            n = nbatches_test // n_test_iter
+            batch_remainder = nbatches_test % n_test_iter
+            
+            while i < n:
+                # load 2 iter
+                if i == 0:
+                    cur_batch_0 = test_ld.dataset[i]
+                    cur_batch_1 = test_ld.dataset[i + 1]
+
+                    X_test0, lS_o_test0, lS_i_test0, T_test0, _, _ = unpack_batch(
+                        cur_batch_0
+                    )
+                    X_test1, lS_o_test1, lS_i_test1, T_test1, _, _ = unpack_batch(
+                        cur_batch_1
+                    )
+                    cur_data[0]['dense_x'] = X_test0
+                    cur_data[0]['T_test'] = T_test0
+                    cur_data[0]['lS_i'] = lS_i_test0
+                    cur_data[0]['lS_o'] = lS_o_test0
+
+                    cur_data[1]['dense_x'] = X_test1
+                    cur_data[1]['T_test'] = T_test1
+                    cur_data[1]['lS_i'] = lS_i_test1
+                    cur_data[1]['lS_o'] = lS_o_test1
+                else:
+                    for j in range(n_test_iter):
+                        cur_data[j] = next_data[j]
+                # preload 2 iter
+                if i + 1 < n:
+                    next_batch_0 = test_ld.dataset[(i+1)*n_test_iter]
+                    next_batch_1 = test_ld.dataset[(i+1)*n_test_iter + 1]
+
+                    X_test0, lS_o_test0, lS_i_test0, T_test0, _, _ = unpack_batch(
+                        next_batch_0
+                    )
+                    X_test1, lS_o_test1, lS_i_test1, T_test1, _, _ = unpack_batch(
+                        next_batch_1
+                    )
+                    next_data[0] = {'dense_x': X_test0, 'T_test': T_test0, 'lS_i': lS_i_test0, 'lS_o': lS_o_test0}
+
+                    next_data[1] = {'dense_x': X_test1, 'T_test': T_test1, 'lS_i': lS_i_test1, 'lS_o': lS_o_test1}
+                elif batch_remainder == 1:
+                    next_batch_0 = test_ld.dataset[len(test_ld) - 1]
+                    X_test0, lS_o_test0, lS_i_test0, T_test0, _, _ = unpack_batch(
+                        next_batch_0
+                    )
+                    next_data[0] = {'dense_x': X_test0, 'T_test': T_test0, 'lS_i': lS_i_test0, 'lS_o': lS_o_test0}
+                    next_data[1] = None
+
+                res_pack = dlrm.distributed_forward_2iter_overlap(cur_data, next_data)
+                if i == n - 1:
+                    last_Z_test = dlrm(next_data[0]['dense_x'], next_data[0]['lS_o'], next_data[0]['lS_i'])
+                if args.print_auc:
+                    for j in range(n_test_iter):
+                        if res_pack[j]:
+                            Z_test, T_test = res_pack[j]
+                            S_test = Z_test.detach().cpu().float()  # numpy array
+                            T_test = T_test.detach().cpu().float()  # numpy array
+                            scores.append(S_test)
+                            targets.append(T_test)
+                    if i == n - 1:
+                        S_test = last_Z_test.detach().cpu().float()  # numpy array
+                        T_test = T_test0.detach().cpu().float()  # numpy array
+                        scores.append(S_test)
+                        targets.append(T_test)
+                i += 1
+            
+        else:
+            for i, testBatch in enumerate(test_ld):
+                should_print = ((i + 1) % args.print_freq == 0 or i + 1 == len(test_ld)) and args.inference_only
+                if should_print:
+                    gT = 1000.0 * total_time / total_iter
+                    print(
+                        "Finished {} it {}/{}, {:.2f} ms/it,".format(
+                            "inference", i + 1, len(test_ld), gT
+                        ),
+                        flush=True,
+                    )
+                    total_time = 0
+                    total_iter = 0
+                # early exit if nbatches was set by the user and was exceeded
+                if args.inference_only and nbatches > 0 and i >= nbatches:
+                    break
+                X_test, lS_o_test, lS_i_test, T_test, W_test, CBPP_test = unpack_batch(
+                    testBatch
+                )
+                # forward pass
+                start = time_wrap()
+                Z_test = dlrm(X_test, lS_o_test, lS_i_test)
+                total_time += (time_wrap() - start)
+                total_iter += 1
+                if args.print_auc:
+                    if ext_dist.my_size > 1:
+                        Z_test = ext_dist.all_gather(Z_test, None)
+                        T_test = ext_dist.all_gather(T_test, None)
+                    S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                    T_test = T_test.detach().cpu().float().numpy()  # numpy array
+                    scores.append(S_test)
+                    targets.append(T_test)
+                elif not args.inference_only:
+                    with record_function("DLRM accuracy compute"):
+                        # compute loss and accuracy
+                        S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                        T_test = T_test.detach().cpu().float().numpy()  # numpy array
+
+                        mbs_test = T_test.shape[0]  # = mini_batch_size except last
+                        A_test = np.sum((np.round(S_test, 0) == T_test).astype(np.uint8))
+
+                        test_accu += A_test
+                        test_samp += mbs_test
+                else:
+                    # do nothing to save time
+                    pass
+    
+    if args.print_auc:
+        with record_function("DLRM mlperf sklearn metrics compute"):
+            scores = np.concatenate(scores, axis=0)
+            targets = np.concatenate(targets, axis=0)
+
+            if args.print_recall_precision_f1_ap:
+                metrics = {
+                    "recall": lambda y_true, y_score: sklearn.metrics.recall_score(
+                        y_true=y_true, y_pred=np.round(y_score)
+                    ),
+                    "precision": lambda y_true, y_score: sklearn.metrics.precision_score(
+                        y_true=y_true, y_pred=np.round(y_score)
+                    ),
+                    "f1": lambda y_true, y_score: sklearn.metrics.f1_score(
+                        y_true=y_true, y_pred=np.round(y_score)
+                    ),
+                    "ap": sklearn.metrics.average_precision_score,
+                }
+        roc_auc, _, accuracy = ipex._C.roc_auc_score_all(torch.Tensor(targets), torch.Tensor(scores))
+        validation_results = {}
+        validation_results["roc_auc"] = roc_auc
+        validation_results["accuracy"] = accuracy
+        acc_test = validation_results["accuracy"]
+        if args.print_recall_precision_f1_ap:
+            for metric_name, metric_function in metrics.items():
+                validation_results[metric_name] = metric_function(targets, scores)
+    elif not args.inference_only:
+        acc_test = test_accu / test_samp
+    else:
+        pass
+
+    model_metrics_dict = {
+        "nepochs": args.nepochs,
+        "nbatches": nbatches,
+        "nbatches_test": nbatches_test,
+    }
+    if not args.inference_only:
+        model_metrics_dict["test_acc"] = acc_test
+
+    if args.print_auc:
+        is_best = validation_results["roc_auc"] > best_auc_test
+        if is_best:
+            best_auc_test = validation_results["roc_auc"]
+            model_metrics_dict["test_auc"] = best_auc_test
+
+        result_fmt_str = (" auc {:.4f}, best auc {:.4f},".format(
+                validation_results["roc_auc"], best_auc_test
+            )
+            + " accuracy {:3.3f} %, best accuracy {:3.3f} %".format(
+                validation_results["accuracy"] * 100, best_acc_test * 100
+            ))
+        if args.print_recall_precision_f1_ap:
+            result_fmt_str =  ("recall {:.4f}, precision {:.4f},".format(
+                validation_results["recall"],
+                validation_results["precision"],
+            )
+            + " f1 {:.4f}, ap {:.4f},".format(
+                validation_results["f1"], validation_results["ap"]
+            )) + result_fmt_str
+        if (ext_dist.my_size > 1 and ext_dist.dist.get_rank() == 1) or ext_dist.my_size <= 1:
+            print(
+                result_fmt_str,
+                flush=True,
+            )
+            print("Accuracy: {:.34} ".format(validation_results["roc_auc"]))
+        if not args.inference_only:
+            if args.mlperf_auc_threshold != 0.0 and best_auc_test > args.mlperf_auc_threshold:
+                train_end = time.time()
+                print("Have reached the auc threshold:", args.mlperf_auc_threshold, ", stop training")
+    elif not args.inference_only:
+        is_best = acc_test > best_acc_test
+        if is_best:
+            best_acc_test = acc_test
+        print(
+            " accuracy {:3.3f} %, best {:3.3f} %".format(
+                acc_test * 100, best_acc_test * 100
+            ),
+            flush=True,
+        )
+    else:
+        pass
+    if not args.inference_only:
+        return model_metrics_dict, is_best
+    else:
+        return
+
+
+def run():
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(
+        description="Train Deep Learning Recommendation Model (DLRM)"
+    )
+    # model related parameters
+    parser.add_argument("--arch-sparse-feature-size", type=int, default=2)
+    parser.add_argument(
+        "--arch-embedding-size", type=dash_separated_ints, default="4-3-2"
+    )
+    # j will be replaced with the table number
+    parser.add_argument("--arch-mlp-bot", type=dash_separated_ints, default="4-3-2")
+    parser.add_argument("--arch-mlp-top", type=dash_separated_ints, default="4-2-1")
+    # activations and loss
+    parser.add_argument("--activation-function", type=str, default="relu")
+    parser.add_argument("--loss-threshold", type=float, default=0.0)  # 1.0e-7
+    parser.add_argument("--round-targets", type=bool, default=False)
+    # data
+    parser.add_argument("--num-batches", type=int, default=0)
+    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
+    parser.add_argument("--raw-data-file", type=str, default="")
+    parser.add_argument("--processed-data-file", type=str, default="")
+    parser.add_argument("--max-ind-range", type=int, default=-1)
+    parser.add_argument("--memory-map", action="store_true", default=False)
+    parser.add_argument("--data-sub-sample-rate", type=float, default=0.0)  # in [0, 1]
+    parser.add_argument("--data-randomize", type=str, default="total")  # or day or none
+    parser.add_argument(
+        "--dataset-multiprocessing",
+        action="store_true",
+        default=False,
+        help="The Kaggle dataset can be multiprocessed in an environment \
+                        with more than 7 CPU cores and more than 20 GB of memory. \n \
+                        The Terabyte dataset can be multiprocessed in an environment \
+                        with more than 24 CPU cores and at least 1 TB of memory.",
+    )
+    # training
+    parser.add_argument("--mini-batch-size", type=int, default=-1)
+    parser.add_argument("--local-batch-size", type=int, default=1)
+    parser.add_argument("--nepochs", type=int, default=1)
+    parser.add_argument("--learning-rate", type=float, default=0.01)
+    parser.add_argument("--print-precision", type=int, default=5)
+    parser.add_argument("--numpy-rand-seed", type=int, default=123)
+    # inference
+    parser.add_argument("--inference-only", action="store_true", default=False)
+    # store/load model
+    parser.add_argument("--save-model", type=str, default="")
+    parser.add_argument("--load-model", type=str, default="")
+    # debugging and profiling
+    parser.add_argument("--print-freq", type=int, default=1)
+    parser.add_argument("--test-freq", type=int, default=6400)
+    parser.add_argument("--test-mini-batch-size", type=int, default=-1)
+    parser.add_argument("--print-time", action="store_true", default=False)
+    parser.add_argument("--print-wall-time", action="store_true", default=False)
+    parser.add_argument("--enable-profiling", action="store_true", default=False)
+    # stop at target AUC Terabyte (no subsampling) 0.8025
+    parser.add_argument("--mlperf-auc-threshold", type=float, default=0.0)
+    parser.add_argument("--mlperf-bin-loader", action="store_true", default=False)
+    parser.add_argument("--mlperf-bin-shuffle", action="store_true", default=False)
+    # LR policy
+    parser.add_argument("--lr-num-warmup-steps", type=int, default=0)
+    parser.add_argument("--lr-decay-start-step", type=int, default=0)
+    parser.add_argument("--lr-num-decay-steps", type=int, default=0)
+    # memrec
+    parser.add_argument("--D", type=int, default=10000)
+    parser.add_argument("--dw", type=int, default=10000)
+    parser.add_argument("--K", type=int, default=9)
+    parser.add_argument("--kw", type=int, default=1)
+    parser.add_argument("--hash-tech-id", type=int, default=2) #2 for wyhash
+    parser.add_argument("--random-codebook-size", type=int, default=100000)
+    parser.add_argument("--arch-mlp-bot-sparse", type=dash_separated_ints, default="512")
+    # intel
+    parser.add_argument("--print-auc", action="store_true", default=False)
+    parser.add_argument("--print-recall-precision-f1-ap", action="store_true", default=False)
+    parser.add_argument("--should-test", action="store_true", default=False)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--bf32", action="store_true", default=False)
+    parser.add_argument("--share-weight-instance", type=int, default=0)
+    parser.add_argument("--num-cpu-cores", type=int, default=0)
+    parser.add_argument("--ipex-interaction", action="store_true", default=False)
+    parser.add_argument("--ipex-merged-emb", action="store_true", default=False)
+    parser.add_argument("--num-warmup-iters", type=int, default=1000)
+    parser.add_argument("--int8", action="store_true", default=False)
+    parser.add_argument("--calibration", action="store_true", default=False)
+    parser.add_argument("--int8-configure", type=str, default="./int8_configure.json")
+    parser.add_argument("--dist-backend", type=str, default="ccl")
+     # embedding table is sparse table only if sparse_dense_boundary >= 2048
+    parser.add_argument("--sparse-dense-boundary", type=int, default=2048)
+    parser.add_argument("--hybrid-gradient-emb", action="store_true", default=False)
+
+    global args
+    global nbatches
+    global nbatches_test
+    global D, dw
+    args = parser.parse_args()
+    D = args.D
+    dw = args.dw
+    dp.initialize_globals(args.D, args.dw, args.K, args.kw, args.random_codebook_size, args.test_mini_batch_size)
+    
+    print(args)
+    ext_dist.init_distributed(backend=args.dist_backend)
+    if args.mini_batch_size < 0:
+        args.mini_batch_size = args.local_batch_size * ext_dist.my_size
+
+    ### some basic setup ###
+    np.random.seed(args.numpy_rand_seed)
+    np.set_printoptions(precision=args.print_precision)
+    torch.set_printoptions(precision=args.print_precision)
+    torch.manual_seed(args.numpy_rand_seed)
+
+    if args.test_mini_batch_size < 0:
+        # if the parameter is not set, use the training batch size
+        args.test_mini_batch_size = args.mini_batch_size
+
+    device = torch.device("cpu")
+    print("Using CPU...")
+
+    ### prepare training data ###
+    ln_bot = np.fromstring(args.arch_mlp_bot, dtype=int, sep="-")
+    arch_mlp_bot_sparse_adjusted = str(D) + "-" + args.arch_mlp_bot_sparse
+    ln_bot_sparse = np.fromstring(arch_mlp_bot_sparse_adjusted, dtype=int, sep="-")
+    # input data
+
+    global train_ld
+    train_data, train_ld, test_data, test_ld = dp.make_criteo_data_and_loaders(args)
+    nbatches = args.num_batches if args.num_batches > 0 else len(train_ld)
+    nbatches_test = len(test_ld)
+
+    ln_emb = train_data.counts
+    # enforce maximum limit on number of vectors per embedding
+    if args.max_ind_range > 0:
+        ln_emb = np.array(
+            list(
+                map(
+                    lambda x: x if x < args.max_ind_range else args.max_ind_range,
+                    ln_emb,
+                )
+            )
+        )
+    else:
+        ln_emb = np.array(ln_emb)
+    m_den = train_data.m_den
+    ln_bot[0] = m_den
+
+    args.ln_emb = ln_emb.tolist()
+
+    ### parse command line arguments ###
+    m_spa = args.arch_sparse_feature_size
+    ln_emb = np.asarray(ln_emb)
+    num_fea = ln_emb.size + 1  # num sparse + num dense features
+
+    m_den_out = ln_bot[ln_bot.size - 1]
+    m_sparse_out = ln_bot_sparse[ln_bot_sparse.size - 1]
+    # approach 1: all
+    # num_int = num_fea * num_fea + m_den_out
+    # approach 2: unique
+
+
+    num_int = m_sparse_out + m_den_out
+
+    #num_int = (num_fea * (num_fea - 1)) // 2 + m_den_out
+
+    arch_mlp_top_adjusted = str(num_int) + "-" + args.arch_mlp_top
+    ln_top = np.fromstring(arch_mlp_top_adjusted, dtype=int, sep="-")
+    arch_mlp_bot_sparse_adjusted = str(D) + "-" + args.arch_mlp_bot_sparse
+    ln_bot_sparse = np.fromstring(arch_mlp_bot_sparse_adjusted, dtype=int, sep="-")
+
+    ### construct the neural network specified above ###
+    # WARNING: to obtain exactly the same initialization for
+    # the weights we need to start from the same random seed.
+    # np.random.seed(args.numpy_rand_seed)
+    global dlrm
+    np_init_emb_weight = not args.inference_only
+    dlrm = DLRM_Net(
+        m_spa,
+        ln_emb,
+        ln_bot,
+        ln_bot_sparse,
+        ln_top,
+        sigmoid_bot=-1,
+        sigmoid_top=ln_top.size - 2,
+        loss_threshold=args.loss_threshold,
+        np_init_emb_weight=np_init_emb_weight,
+        K=args.K,
+        kw=args.kw
+    )
+    if args.ipex_merged_emb:
+        if ext_dist.my_size > 1:
+            dlrm.emb_sparse = ipex.nn.modules.MergedEmbeddingBagWithSGD.from_embeddingbag_list(dlrm.emb_sparse, lr=args.learning_rate/ext_dist.my_size)
+            dlrm.emb_dense = ipex.nn.modules.MergedEmbeddingBag.from_embeddingbag_list(dlrm.emb_dense)
+            dlrm.need_linearize_indices_and_offsets = torch.BoolTensor([False])
+        else:
+            dlrm.emb_l = ipex.nn.modules.MergedEmbeddingBagWithSGD.from_embeddingbag_list(dlrm.emb_l, lr=args.learning_rate)
+            dlrm.need_linearize_indices_and_offsets = torch.BoolTensor([False])
+
+    if not args.inference_only:
+        #ToDo: may cause convergence issue for sparse table lr
+        optimizer = torch.optim.SGD(dlrm.parameters(), lr=args.learning_rate)
+        lr_scheduler = LRPolicyScheduler(
+            optimizer,
+            args.lr_num_warmup_steps,
+            args.lr_decay_start_step,
+            args.lr_num_decay_steps,
+        )
+
+    ### main loop ###
+    # training or inference
+    best_acc_test = 0
+    best_auc_test = 0
+    skip_upto_epoch = 0
+    skip_upto_batch = 0
+    total_time = 0
+    total_loss = 0
+    total_iter = 0
+    total_samp = 0
+
+    # Load model is specified
+    if not (args.load_model == ""):
+        print("Loading saved model {}".format(args.load_model))
+        print("Loading weight will minimally cost ~{} G memory".format(dlrm.numel * 4 / 1024 / 1024 / 1024))
+        print("Current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        ld_model = torch.load(args.load_model, map_location=torch.device("cpu"))
+        print("Loading weight done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        dlrm.load_state_dict(ld_model["state_dict"])
+        ld_j = ld_model["iter"]
+        ld_k = ld_model["epoch"]
+        ld_nepochs = ld_model["nepochs"]
+        ld_nbatches = ld_model["nbatches"]
+        ld_nbatches_test = ld_model["nbatches_test"]
+        ld_train_loss = ld_model["train_loss"]
+        ld_total_loss = ld_model["total_loss"]
+        ld_acc_test = ld_model["test_acc"]
+        if not args.inference_only:
+            optimizer.load_state_dict(ld_model["opt_state_dict"])
+            best_acc_test = ld_acc_test
+            total_loss = ld_total_loss
+            skip_upto_epoch = ld_k  # epochs
+            skip_upto_batch = ld_j  # batches
+        else:
+            if (args.num_batches >= nbatches_test) or (args.num_batches == 0):
+                nbatches = nbatches_test-1
+            args.test_freq = 0
+        print("Copy loaded state to model done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        del(ld_model)
+        print("Remove loaded state, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        print(
+            "Saved at: epoch = {:d}/{:d}, batch = {:d}/{:d}, ntbatch = {:d}".format(
+                ld_k, ld_nepochs, ld_j, ld_nbatches, ld_nbatches_test
+            )
+        )
+        print(
+            "Training state: loss = {:.6f}".format(
+                ld_train_loss,
+            )
+        )
+        print("Testing state: accuracy = {:3.3f} %".format(ld_acc_test * 100))
+
+    ext_dist.barrier()
+
+    if args.calibration:
+        assert args.load_model != "", "need load weight to do calibration"
+        dlrm.eval()
+        qconfig = QConfig(activation=MinMaxObserver.with_args(qscheme=torch.per_tensor_symmetric, dtype=torch.qint8),
+            weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric))
+        for j, inputBatch in enumerate(train_ld):
+            X, lS_o, lS_i, T, W, CBPP = unpack_batch(inputBatch)
+            example_inputs = (X, lS_o, lS_i)
+            prepared_dlrm = prepare(dlrm, qconfig, example_inputs=example_inputs, inplace=True)
+            break
+
+        for j, inputBatch in enumerate(train_ld):
+            prepared_dlrm(X, lS_o, lS_i)
+            if j == 64:
+                break
+        prepared_dlrm.save_qconf_summary(qconf_summary = args.int8_configure)
+        print("calibration done, save config file to ", args.int8_configure)
+        exit()
+
+    print("time/loss/accuracy (if enabled):")
+    
+    global buffer_num
+    if ext_dist.my_size > 1:
+        buffer_num = 1
+    global data_buffer
+    data_buffer = buffer_num * [None]
+    global data_iter
+    data_iter = iter(train_ld)
+    buffer_num = buffer_num if buffer_num <= nbatches else nbatches
+    # data_load_begin = time.time()
+    load_data(buffer_num, args.bf16)
+    # print(buffer_num, ": data item loaded, data_load_time is {:.6f}s".format(time.time() - data_load_begin))
+
+        
+    if not args.inference_only:
+        X, lS_o, lS_i, T = train_ld.dataset.__getitem__(0)
+        #if ext_dist.my_size > 1:
+        #    local_bs = X.size()[0] // ext_dist.my_size
+        #    rank_id = dlrm.rank
+        #    X = X[rank_id * local_bs: (rank_id + 1) * local_bs]
+        #    T = T[rank_id * local_bs: (rank_id + 1) * local_bs]
+        #    global_bs = local_bs * ext_dist.my_size
+        #    lS_o_sparse = lS_o[dlrm.local_ln_emb_sparse, :global_bs]
+        #    lS_i_sparse = lS_i[dlrm.local_ln_emb_sparse, :global_bs]
+        #    lS_o_dense = lS_o[dlrm.local_ln_emb_sparse, rank_id * local_bs: (rank_id + 1) * local_bs]
+        #    lS_i_dense = lS_i[dlrm.local_ln_emb_sparse, rank_id * local_bs: (rank_id + 1) * local_bs]
+
+        #if isinstance(dlrm.emb_dense, ipex.nn.modules.MergedEmbeddingBag) or isinstance(dlrm.emb_l, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+        #    if ext_dist.my_size > 1:#use hybrid gradient merged embedding 16 sparse/ 10 dense 
+        #        batch_size = X.size()[0]
+        #        n_tables_sparse = lS_i_sparse.shape[0]
+        #        #prepare the input for sparse mergedembedding
+        #        idx_sparse = [lS_i_sparse[i] for i in range(n_tables)]
+        #        offset_sparse = [lS_o_sparse[i] for i in range(n_tables)]
+        #        include_last = [False for i in range(n_tables)]
+        #        indices, offsets, indices_with_row_offsets = dlrm.emb_l.linearize_indices_and_offsets(idx_sparse, offset_sparse, include_last_sparse)
+        #        #prepare the input for dense mergedembedding 
+        #        n_tables_dense = lS_i_dense.shape[0]
+        #        idx_dense = [lS_i_dense[i] for i in range(n_tables)]
+        #        offset_dense = [lS_o_dense[i] for i in range(n_tables)]
+        #        include_last = [False for i in range(n_tables)]
+        #        indices, offsets, indices_with_row_offsets = dlrm.emb_l.linearize_indices_and_offsets(idx_dense, offset_dense, include_last_dense)
+        #    else:
+        #        n_tables = lS_i.shape[0]
+        #        idx = [lS_i[i] for i in range(n_tables)]
+        #        offset = [lS_o[i] for i in range(n_tables)]
+        #        include_last = [False for i in range(n_tables)]
+        #        indices, offsets, indices_with_row_offsets = dlrm.emb_l.linearize_indices_and_offsets(idx, offset, include_last)
+        #if isinstance(dlrm.emb_l, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+        #    sample_input = (X, indices, offsets, indices_with_row_offsets)
+        #else:
+        sample_input = (X, lS_o, lS_i)
+        if args.bf16:
+            print("Start to split weight to bf16 and trail part, or saving whole fp32 master weight, create bf16 weight copy")
+            print("Maximum will use ~ {} G memory, may use less memory if useless fp32 weight (for split path) will be released in time ".format(dlrm.numel * 4 / 1024 / 1024 /1024))
+            dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.bfloat16, optimizer=optimizer, inplace=True, sample_input=sample_input)
+            print("args.ipex_merged_emb:", args.ipex_merged_emb)
+            if args.ipex_merged_emb:
+                print("###############Current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+                if ext_dist.my_size > 1:
+                    dlrm.emb_sparse.to_bfloat16_train()
+                else:
+                    dlrm.emb_l.to_bfloat16_train()
+            print("Weight cast done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+        else:
+            dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.float, optimizer=optimizer, inplace=True, sample_input=sample_input, auto_kernel_selection=True)
+            if args.bf32:
+                ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+
+        for i in range(len(dlrm.top_l)):
+            if isinstance(dlrm.top_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
+                if isinstance(dlrm.top_l[i+1], torch.nn.ReLU):
+                    dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'relu')
+                else:
+                    dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'sigmoid')
+                dlrm.top_l[i + 1] = torch.nn.Identity()
+        for i in range(len(dlrm.bot_l)):
+            if isinstance(dlrm.bot_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
+                if isinstance(dlrm.bot_l[i+1], torch.nn.ReLU):
+                    dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'relu')
+                else:
+                    dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'sigmoid')
+                dlrm.bot_l[i + 1] = torch.nn.Identity()
+
+        if ext_dist.my_size > 1:
+            dlrm.bot_l = ext_dist.DDP(dlrm.bot_l, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
+            dlrm.top_l = ext_dist.DDP(dlrm.top_l, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
+            dlrm.emb_dense = ext_dist.DDP(dlrm.emb_dense, gradient_as_bucket_view=True, broadcast_buffers=False)
+    training_record = [0, 0]
+    def update_training_performance(time, iters, training_record=training_record):
+        if iters > args.num_warmup_iters:
+            training_record[0] += time
+            training_record[1] += 1
+
+    def print_training_performance( training_record=training_record):
+        if training_record[0] == 0:
+            print("num-batches larger than warm up iters, please increase num-batches or decrease warmup iters")
+            exit()
+        total_samples = training_record[1] * args.mini_batch_size
+        throughput = total_samples / training_record[0] * 1000
+        print("Throughput: {:.3f} fps".format(throughput))
+
+    test_freq = args.test_freq if args.test_freq != -1  else nbatches // 20
+    print("Initialize for not inference only done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
+    # print(dlrm)
+    train_start = time.time()
+    total_train_time_wo_dl_eval = 0
+    wait_it = 0
+    warmup_it = 400
+    active_it = 20
+    lrs =[]
+
+
+    def trace_handler(prof):
+        print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+        if ext_dist.my_size > 1:
+            rank = ext_dist.dist.get_rank()
+            prof.export_chrome_trace("dlrm_training_trace_rank_{}_step{}.json".format(rank,str(prof.step_num)))
+        else:
+            rank = 0
+            prof.export_chrome_trace("dlrm_training_trace_step_{}.json".format(str(prof.step_num)))
+
+        file_prefix = "%s/dlrm_s_pytorch_r%d" % (".", rank)
+        #with open("dlrm_s_pytorch.prof", "w") as prof_f:
+        with open("%s.prof" % file_prefix, "w") as prof_f:
+            prof_f.write(prof.key_averages().table(sort_by="cpu_time_total"))
+
+    with torch.profiler.profile(
+        activities=[ProfilerActivity.CPU],
+        schedule=torch.profiler.schedule(
+            wait=wait_it,
+            warmup=warmup_it,
+            active=active_it),
+        on_trace_ready=trace_handler
+        ) as prof:
+        if not args.inference_only:
+            k = 0
+            while k < args.nepochs:
+
+                if k < skip_upto_epoch:
+                    continue
+                
+                j = 0
+                # for j, inputBatch in enumerate(train_ld):
+                while j < nbatches: 
+                    # if j < skip_upto_batch:
+                    #     continue
+
+                    for d in range(buffer_num):
+                        # X, lS_o, lS_i, T, W, CBPP = unpack_batch(inputBatch)
+                        (X, lS_o, lS_i, T, W, CBPP) = data_buffer[d]
+                        
+                        t1 = time_wrap()
+
+                        # early exit if nbatches was set by the user and has been exceeded
+                        if nbatches > 0 and j >= nbatches:
+                            break
+
+                        mbs = T.shape[0]  # = args.mini_batch_size except maybe for last
+
+                        # forward pass
+
+                        if hasattr(dlrm, 'emb_l') and isinstance(dlrm.emb_l, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+                            dlrm.emb_l.sgd_args = dlrm.emb_l.sgd_args._replace(lr=lr_scheduler.get_last_lr()[0])
+
+                        if hasattr(dlrm, 'emb_sparse') and isinstance(dlrm.emb_sparse, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+                            dlrm.emb_sparse.sgd_args = dlrm.emb_sparse.sgd_args._replace(lr=lr_scheduler.get_last_lr()[0]/ext_dist.my_size)
+
+                        with torch.cpu.amp.autocast(enabled=args.bf16):
+                            Z = dlrm_wrap(
+                                X,
+                                lS_o,
+                                lS_i,
+                                True,
+                            ).float()
+                        # losis
+                        E = loss_fn_wrap(Z, T)
+
+                        # compute loss and accuracy
+                        L = E.detach().cpu().numpy()  # numpy array
+
+                        with record_function("DLRM backward"):
+                            # scaled error gradient propagation
+                            # (where we do not accumulate gradients across mini-batches)
+                            optimizer.zero_grad(set_to_none=True)
+                            # backward pass
+                            E.backward()
+                        with record_function("DLRM update"):
+                            # optimizer
+                            optimizer.step()
+
+                        lr_scheduler.step()
+
+                        t2 = time_wrap() - dlrm.emb_args_preprocessing
+                        dlrm.emb_args_preprocessing = 0 
+                        total_train_time_wo_dl_eval += (t2 - t1)
+                        total_time += t2 - t1
+
+                        total_loss += L * mbs
+                        total_iter += 1
+                        total_samp += mbs
+                        if args.enable_profiling:
+                            prof.step()
+                            if j >=  wait_it + warmup_it + active_it:
+                                break
+                        should_print = ((j + 1) % args.print_freq == 0) or (
+                            j + 1 == nbatches
+                        )
+                        should_test = (
+                            (args.should_test)
+                            and (((j + 1) % test_freq == 0) or (j + 1 == nbatches))
+                        )
+
+                        # print time, loss and accuracy
+                        if should_print or should_test:
+                            gT = 1000.0 * total_time / total_iter if args.print_time else -1
+                            total_time = 0
+
+                            train_loss = total_loss / total_samp
+                            total_loss = 0
+
+                            str_run_type = (
+                                "inference" if args.inference_only else "training"
+                            )
+
+                            wall_time = ""
+                            if args.print_wall_time:
+                                wall_time = " ({})".format(time.strftime("%H:%M"))
+                            # if ext_dist.my_size > 1 and ext_dist.dist.get_rank() ==1 :
+                            print(
+                                "Finished {} it {}/{} of epoch {}, {:.2f} ms/it,".format(
+                                    str_run_type, j + 1, nbatches, k, gT
+                                )
+                                + " loss {:.6f}".format(train_loss)
+                                + wall_time,
+                                flush=True,
+                            )
+                            update_training_performance(gT, j)
+
+                            total_iter = 0
+                            total_samp = 0
+
+                        # testing
+                        if should_test:
+                            eval_begin = time.time()
+                            model_metrics_dict, is_best = inference(
+                                args,
+                                dlrm,
+                                best_acc_test,
+                                best_auc_test,
+                                test_ld,
+                            )
+                            if is_best:
+                                best_auc_test = model_metrics_dict["test_auc"]
+                                best_acc_test = model_metrics_dict["test_acc"]
+                            eval_end = time.time()
+                            dlrm.emb_args_preprocessing = 0
+                            if ext_dist.my_size > 1 and ext_dist.dist.get_rank() ==1 :
+                                print("Evauation at {} iteration using {} s".format(j+1, eval_end- eval_begin))
+                            if (
+                                is_best
+                                and not (args.save_model == "")
+                                and not args.inference_only
+                            ):
+                                model_metrics_dict["epoch"] = k
+                                model_metrics_dict["iter"] = j + 1
+                                model_metrics_dict["train_loss"] = train_loss
+                                model_metrics_dict["total_loss"] = total_loss
+                                model_metrics_dict[
+                                    "opt_state_dict"
+                                ] = optimizer.state_dict()
+                                print("Saving model to {}".format(args.save_model))
+                                torch.save(model_metrics_dict, args.save_model)
+
+                            if (
+                                (args.mlperf_auc_threshold > 0)
+                                and (best_auc_test > args.mlperf_auc_threshold)
+                            ):
+                                train_end = time.time()
+                                if ext_dist.dist.get_rank() == 1:
+                                    print("The TTT w/ dataloader and evaluation is {} mins".format((train_end - train_start)/60.0))
+                                    print("The TTT w/o dataloader and evaluation is {} mins".format((total_train_time_wo_dl_eval)/60.0))
+                                exit()
+                        j += 1
+
+                    if ext_dist.my_size == 1:
+                        buffer_num = buffer_num if (nbatches - j) > buffer_num else (nbatches - j)
+                        next_epoch = buffer_num == 0 or load_data(buffer_num, args.bf16)
+                        if next_epoch:
+                            k += 1
+                            if k >= args.nepochs:
+                                break
+
+                k += 1  # nepochs
+        else:
+            print("Testing for inference only")
+            with torch.no_grad():
+                inference(
+                    args,
+                    dlrm,
+                    best_acc_test,
+                    best_auc_test,
+                    test_ld
+                )
+
+    # profiling
+    # print_training_performance()
+
+    if args.enable_profiling:
+        time_stamp = str(datetime.datetime.now()).replace(" ", "_")
+        with open("dlrm_s_pytorch" + time_stamp + "_shape.prof", "w") as prof_f:
+            prof_f.write(
+                prof.key_averages(group_by_input_shape=True).table(
+                    sort_by="self_cpu_time_total"
+                )
+            )
+        with open("dlrm_s_pytorch" + time_stamp + "_total.prof", "w") as prof_f:
+            prof_f.write(prof.key_averages().table(sort_by="self_cpu_time_total"))
+        prof.export_chrome_trace("dlrm_s_pytorch" + time_stamp + ".json")
+
+if __name__ == "__main__":
+    print("----------MEMREC---------")
+    run()
diff --git a/models/recommendation/pytorch/memrec_dlrm/product/memrec_data.py b/models/recommendation/pytorch/memrec_dlrm/product/memrec_data.py
new file mode 100644
index 000000000..8d4461a96
--- /dev/null
+++ b/models/recommendation/pytorch/memrec_dlrm/product/memrec_data.py
@@ -0,0 +1,628 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Facebook, Inc. and its affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+# Description: generate inputs and targets for the dlrm benchmark
+# The inpts and outputs are generated according to the following three option(s)
+# 1) random distribution
+# 2) synthetic distribution, based on unique accesses and distances between them
+#    i) R. Hassan, A. Harris, N. Topham and A. Efthymiou "Synthetic Trace-Driven
+#    Simulation of Cache Memory", IEEE AINAM'07
+# 3) public data set
+#    i)  Criteo Kaggle Display Advertising Challenge Dataset
+#    https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
+#    ii) Criteo Terabyte Dataset
+#    https://labs.criteo.com/2013/12/download-terabyte-click-logs
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+# others
+from os import path
+import sys
+
+import data_utils
+import bisect
+import collections
+import memrec_encoder
+
+# numpy
+import numpy as np
+from numpy import random as ra
+
+
+# pytorch
+import torch
+from torch.utils.data import Dataset, RandomSampler
+
+import data_loader_terabyte
+import random
+import string
+from tqdm import tqdm
+
+
+def initialize_globals(D_value, dw_value, K_value, kw_value, codebook_size, bs):
+    global random_codebook, D, dw, K, kw, lS_o_preconstructed
+    D = D_value
+    dw = dw_value
+    K = K_value
+    kw = kw_value
+    num_ones_te = K_value*26
+    num_ones_we = kw_value*26
+    lS_o_K = [i*num_ones_te for i in range(bs)]
+    lS_o_kw = [i*num_ones_we for i in range(bs)]
+    lS_o_preconstructed = torch.tensor(lS_o_K + lS_o_kw, dtype=torch.int32)
+
+# Kaggle Display Advertising Challenge Dataset
+# dataset (str): name of dataset (Kaggle or Terabyte)
+# randomize (str): determines randomization scheme
+#            "none": no randomization
+#            "day": randomizes each day"s data (only works if split = True)
+#            "total": randomizes total dataset
+# split (bool) : to split into train, test, validation data-sets
+class CriteoDataset(Dataset):
+
+    def __init__(
+            self,
+            dataset,
+            max_ind_range,
+            sub_sample_rate,
+            randomize,
+            split="train",
+            raw_path="",
+            pro_data="",
+            memory_map=False,
+            dataset_multiprocessing=False,
+    ):
+        # dataset
+        # tar_fea = 1   # single target
+        den_fea = 13  # 13 dense  features
+        # spa_fea = 26  # 26 sparse features
+        # tad_fea = tar_fea + den_fea
+        # tot_fea = tad_fea + spa_fea
+        if dataset == "kaggle":
+            days = 7
+            out_file = "kaggleAdDisplayChallenge_processed"
+        elif dataset == "terabyte":
+            days = 24
+            out_file = "terabyte_processed"
+        else:
+            raise(ValueError("Data set option is not supported"))
+        self.max_ind_range = max_ind_range
+        self.memory_map = memory_map
+
+        # split the datafile into path and filename
+        lstr = raw_path.split("/")
+        self.d_path = "/".join(lstr[0:-1]) + "/"
+        self.d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+        self.npzfile = self.d_path + (
+            (self.d_file + "_day") if dataset == "kaggle" else self.d_file
+        )
+        self.trafile = self.d_path + (
+            (self.d_file + "_fea") if dataset == "kaggle" else "fea"
+        )
+
+        # check if pre-processed data is available
+        data_ready = True
+        if memory_map:
+            for i in range(days):
+                reo_data = self.npzfile + "_{0}_reordered.npz".format(i)
+                if not path.exists(str(reo_data)):
+                    data_ready = False
+        else:
+            if not path.exists(str(pro_data)):
+                data_ready = False
+
+        # pre-process data if needed
+        # WARNNING: when memory mapping is used we get a collection of files
+        if data_ready:
+            print("Reading pre-processed data=%s" % (str(pro_data)))
+            file = str(pro_data)
+        else:
+            print("Reading raw data=%s" % (str(raw_path)))
+            file = data_utils.getCriteoAdData(
+                raw_path,
+                out_file,
+                max_ind_range,
+                sub_sample_rate,
+                days,
+                split,
+                randomize,
+                dataset == "kaggle",
+                memory_map,
+                dataset_multiprocessing,
+            )
+
+        # get a number of samples per day
+        total_file = self.d_path + self.d_file + "_day_count.npz"
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"]
+        # compute offsets per file
+        self.offset_per_file = np.array([0] + [x for x in total_per_file])
+        for i in range(days):
+            self.offset_per_file[i + 1] += self.offset_per_file[i]
+        # print(self.offset_per_file)
+
+        # setup data
+        if memory_map:
+            # setup the training/testing split
+            self.split = split
+            if split == 'none' or split == 'train':
+                self.day = 0
+                self.max_day_range = days if split == 'none' else days - 1
+            elif split == 'test' or split == 'val':
+                self.day = days - 1
+                num_samples = self.offset_per_file[days] - \
+                              self.offset_per_file[days - 1]
+                self.test_size = int(np.ceil(num_samples / 2.))
+                self.val_size = num_samples - self.test_size
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train or test.")
+
+            '''
+            # text
+            print("text")
+            for i in range(days):
+                fi = self.npzfile + "_{0}".format(i)
+                with open(fi) as data:
+                    ttt = 0; nnn = 0
+                    for _j, line in enumerate(data):
+                        ttt +=1
+                        if np.int32(line[0]) > 0:
+                            nnn +=1
+                    print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                          + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            # processed
+            print("processed")
+            for i in range(days):
+                fi = self.npzfile + "_{0}_processed.npz".format(i)
+                with np.load(fi) as data:
+                    yyy = data["y"]
+                ttt = len(yyy)
+                nnn = np.count_nonzero(yyy)
+                print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                      + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            # reordered
+            print("reordered")
+            for i in range(days):
+                fi = self.npzfile + "_{0}_reordered.npz".format(i)
+                with np.load(fi) as data:
+                    yyy = data["y"]
+                ttt = len(yyy)
+                nnn = np.count_nonzero(yyy)
+                print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                      + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            '''
+
+            # load unique counts
+            with np.load(self.d_path + self.d_file + "_fea_count.npz") as data:
+                self.counts = data["counts"]
+            self.m_den = den_fea  # X_int.shape[1]
+            self.n_emb = len(self.counts)
+            print("Sparse features= %d, Dense features= %d" % (self.n_emb, self.m_den))
+
+            # Load the test data
+            # Only a single day is used for testing
+            if self.split == 'test' or self.split == 'val':
+                # only a single day is used for testing
+                fi = self.npzfile + "_{0}_reordered.npz".format(
+                    self.day
+                )
+                with np.load(fi) as data:
+                    self.X_int = data["X_int"]  # continuous  feature
+                    self.X_cat = data["X_cat"]  # categorical feature
+                    self.y = data["y"]          # target
+
+        else:
+            # load and preprocess data
+            with np.load(file) as data:
+                X_int = data["X_int"]  # continuous  feature
+                X_cat = data["X_cat"]  # categorical feature
+                y = data["y"]          # target
+                self.counts = data["counts"]
+            self.m_den = X_int.shape[1]  # den_fea
+            self.n_emb = len(self.counts)
+            print("Sparse fea = %d, Dense fea = %d" % (self.n_emb, self.m_den))
+
+            # create reordering
+            indices = np.arange(len(y))
+
+            if split == "none":
+                # randomize all data
+                if randomize == "total":
+                    indices = np.random.permutation(indices)
+                    print("Randomized indices...")
+
+                X_int[indices] = X_int
+                X_cat[indices] = X_cat
+                y[indices] = y
+
+            else:
+                indices = np.array_split(indices, self.offset_per_file[1:-1])
+
+                # randomize train data (per day)
+                if randomize == "day":  # or randomize == "total":
+                    for i in range(len(indices) - 1):
+                        indices[i] = np.random.permutation(indices[i])
+                    print("Randomized indices per day ...")
+
+                train_indices = np.concatenate(indices[:-1])
+                test_indices = indices[-1]
+                test_indices, val_indices = np.array_split(test_indices, 2)
+
+                print("Defined %s indices..." % (split))
+
+                # randomize train data (across days)
+                if randomize == "total":
+                    train_indices = np.random.permutation(train_indices)
+                    print("Randomized indices across days ...")
+
+                # create training, validation, and test sets
+                if split == 'train':
+                    self.X_int = [X_int[i] for i in train_indices]
+                    self.X_cat = [X_cat[i] for i in train_indices]
+                    self.y = [y[i] for i in train_indices]
+                elif split == 'val':
+                    self.X_int = [X_int[i] for i in val_indices]
+                    self.X_cat = [X_cat[i] for i in val_indices]
+                    self.y = [y[i] for i in val_indices]
+                elif split == 'test':
+                    self.X_int = [X_int[i] for i in test_indices]
+                    self.X_cat = [X_cat[i] for i in test_indices]
+                    self.y = [y[i] for i in test_indices]
+
+            print("Split data according to indices...")
+
+    def __getitem__(self, index):
+
+        if isinstance(index, slice):
+            return [
+                self[idx] for idx in range(
+                    index.start or 0, index.stop or len(self), index.step or 1
+                )
+            ]
+
+        if self.memory_map:
+            if self.split == 'none' or self.split == 'train':
+                # check if need to swicth to next day and load data
+                if index == self.offset_per_file[self.day]:
+                    # print("day_boundary switch", index)
+                    self.day_boundary = self.offset_per_file[self.day]
+                    fi = self.npzfile + "_{0}_reordered.npz".format(
+                        self.day
+                    )
+                    # print('Loading file: ', fi)
+                    with np.load(fi) as data:
+                        self.X_int = data["X_int"]  # continuous  feature
+                        self.X_cat = data["X_cat"]  # categorical feature
+                        self.y = data["y"]          # target
+                    self.day = (self.day + 1) % self.max_day_range
+
+                i = index - self.day_boundary
+            elif self.split == 'test' or self.split == 'val':
+                # only a single day is used for testing
+                i = index + (0 if self.split == 'test' else self.test_size)
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train or test.")
+        else:
+            i = index
+
+        if self.max_ind_range > 0:
+            return self.X_int[i], self.X_cat[i] % self.max_ind_range, self.y[i]
+        else:
+            return self.X_int[i], self.X_cat[i], self.y[i]
+
+    def _default_preprocess(self, X_int, X_cat, y):
+        X_int = torch.log(torch.tensor(X_int, dtype=torch.float) + 1)
+        if self.max_ind_range > 0:
+            X_cat = torch.tensor(X_cat % self.max_ind_range, dtype=torch.long)
+        else:
+            X_cat = torch.tensor(X_cat, dtype=torch.long)
+        y = torch.tensor(y.astype(np.float32))
+
+        return X_int, X_cat, y
+
+    def __len__(self):
+        if self.memory_map:
+            if self.split == 'none':
+                return self.offset_per_file[-1]
+            elif self.split == 'train':
+                return self.offset_per_file[-2]
+            elif self.split == 'test':
+                return self.test_size
+            elif self.split == 'val':
+                return self.val_size
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train nor test.")
+        else:
+            return len(self.y)
+
+# memrec - collate_wrapper_criteo_offset_tb
+def collate_wrapper_criteo_offset_tb(list_of_tuples, args):
+    # where each tuple is (X_int, X_cat, y)
+    X_int = list_of_tuples[0]
+    #lS_o = torch.transpose(list_of_tuples[1], 0, 1)
+    lS_i = torch.transpose(list_of_tuples[2], 0, 1)
+    T = list_of_tuples[3]
+    batchSize = lS_i.shape[0]
+    featureCnt = lS_i.shape[1]
+    num_ones_te=(K*featureCnt)
+    num_ones_we=(kw*featureCnt)
+
+    hd_encoded_cat_data_ = memrec_encoder.wyh_hash_array_sparse(lS_i.detach().cpu().numpy(), args.D, 8, featureCnt, args.K, args.kw, args.hash_tech_id)
+    lS_i = torch.tensor(hd_encoded_cat_data_, dtype=torch.int32)
+    
+    if (args.inference_only):
+        #lS_o =  lS_o_preconstructed
+        lS_o_K = [i*num_ones_te for i in range(batchSize)]
+        lS_o_kw = [i*num_ones_we for i in range(batchSize)]  
+        lS_o = torch.tensor(lS_o_K + lS_o_kw, dtype=torch.int32)
+    else:
+        lS_o_K = [i*num_ones_te for i in range(batchSize)]
+        lS_o_kw = [i*num_ones_we for i in range(batchSize)]  
+        lS_o = torch.tensor(lS_o_K + lS_o_kw, dtype=torch.int32)
+    #print(lS_o.shape, lS_i.shape)
+    return X_int, lS_o, lS_i, T
+
+# memrec - collate_wrapper_criteo_offset
+def collate_wrapper_criteo_offset(list_of_tuples, args):
+    # where each tuple is (X_int, X_cat, y)
+    transposed_data = list(zip(*list_of_tuples))
+    X_int = torch.log(torch.tensor(np.array(transposed_data[0]), dtype=torch.float) + 1)
+    X_cat = torch.tensor(np.array(transposed_data[1]), dtype=torch.long)
+
+    batchSize = X_cat.shape[0]
+    featureCnt = X_cat.shape[1]
+
+    num_ones_te=(K*featureCnt)
+    num_ones_we=(kw*featureCnt)
+
+    hd_encoded_cat_data_ = memrec_encoder.wyh_hash_array_sparse(X_cat.detach().cpu().numpy(), args.D, 8, featureCnt, args.K, args.kw, args.hash_tech_id)
+    lS_i = torch.tensor(hd_encoded_cat_data_, dtype=torch.int32)
+    T = torch.tensor(np.array(transposed_data[2]), dtype=torch.float32).view(-1, 1)
+    
+    lS_o_K = [i*num_ones_te for i in range(batchSize)]
+    lS_o_kw = [i*num_ones_we for i in range(batchSize)]  
+    lS_o = torch.tensor(lS_o_K + lS_o_kw, dtype=torch.int32)
+
+    return X_int, lS_o, lS_i, T
+
+
+def ensure_dataset_preprocessed(args, d_path):
+    _ = CriteoDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "train",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map,
+        args.dataset_multiprocessing
+    )
+
+    _ = CriteoDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "test",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map,
+        args.dataset_multiprocessing
+    )
+
+    for split in ['train', 'val', 'test']:
+        print('Running preprocessing for split =', split)
+
+        train_files = ['{}_{}_reordered.npz'.format(args.raw_data_file, day)
+                       for
+                       day in range(0, 23)]
+
+        test_valid_file = args.raw_data_file + '_23_reordered.npz'
+
+        output_file = d_path + '_{}.bin'.format(split)
+
+        input_files = train_files if split == 'train' else [test_valid_file]
+        data_loader_terabyte.numpy_to_binary(input_files=input_files,
+                                             output_file_path=output_file,
+                                             split=split)
+
+
+# Conversion from offset to length
+def offset_to_length_converter(lS_o, lS_i):
+    def diff(tensor):
+        return tensor[1:] - tensor[:-1]
+
+    return torch.stack(
+        [
+            diff(torch.cat((S_o, torch.tensor(lS_i[ind].shape))).int())
+            for ind, S_o in enumerate(lS_o)
+        ]
+    )
+
+
+def collate_wrapper_criteo_length(list_of_tuples):
+    # where each tuple is (X_int, X_cat, y)
+    transposed_data = list(zip(*list_of_tuples))
+    X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
+    X_cat = torch.tensor(transposed_data[1], dtype=torch.long)
+    T = torch.tensor(transposed_data[2], dtype=torch.float32).view(-1, 1)
+
+    batchSize = X_cat.shape[0]
+    featureCnt = X_cat.shape[1]
+
+    lS_i = torch.stack([X_cat[:, i] for i in range(featureCnt)])
+    lS_o = torch.stack(
+        [torch.tensor(range(batchSize)) for _ in range(featureCnt)]
+    )
+
+    lS_l = offset_to_length_converter(lS_o, lS_i)
+
+    return X_int, lS_l, lS_i, T
+
+
+def make_criteo_data_and_loaders(args, offset_to_length_converter=False):
+    if args.memory_map and args.data_set == "terabyte":
+        # more efficient for larger batches
+        data_directory = path.dirname(args.raw_data_file)
+        collate_wrapper_criteo = collate_wrapper_criteo_offset_tb
+
+        if args.mlperf_bin_loader:
+            lstr = args.processed_data_file.split("/")
+            d_path = "/".join(lstr[0:-1]) + "/" + lstr[-1].split(".")[0]
+            train_file = d_path + "_train.bin"
+            test_file = d_path + "_test.bin"
+            # val_file = d_path + "_val.bin"
+            counts_file = args.raw_data_file + '_fea_count.npz'
+            if any(not path.exists(p) for p in [train_file,
+                                                test_file,
+                                                counts_file]):
+                ensure_dataset_preprocessed(args, d_path)
+
+            train_data = data_loader_terabyte.CriteoBinDataset(
+                data_file=train_file,
+                counts_file=counts_file,
+                batch_size=args.mini_batch_size,
+                max_ind_range=args.max_ind_range
+            )
+
+            train_loader = torch.utils.data.DataLoader(
+                train_data,
+                batch_size=None,
+                batch_sampler=None,
+                shuffle=False,
+                num_workers=0,
+                collate_fn=lambda b: collate_wrapper_criteo(b, args),
+                pin_memory=False,
+                drop_last=False,
+                sampler=RandomSampler(train_data) if args.mlperf_bin_shuffle else None
+            )
+
+            test_data = data_loader_terabyte.CriteoBinDataset(
+                data_file=test_file,
+                counts_file=counts_file,
+                batch_size=args.test_mini_batch_size,
+                max_ind_range=args.max_ind_range
+            )
+
+            test_loader = torch.utils.data.DataLoader(
+                test_data,
+                batch_size=None,
+                batch_sampler=None,
+                shuffle=False,
+                num_workers=0,
+                collate_fn=lambda b: collate_wrapper_criteo(b, args),
+                pin_memory=False,
+                drop_last=False,
+            )
+        else:
+            data_filename = args.raw_data_file.split("/")[-1]
+
+            train_data = CriteoDataset(
+                args.data_set,
+                args.max_ind_range,
+                args.data_sub_sample_rate,
+                args.data_randomize,
+                "train",
+                args.raw_data_file,
+                args.processed_data_file,
+                args.memory_map,
+                args.dataset_multiprocessing
+            )
+
+            test_data = CriteoDataset(
+                args.data_set,
+                args.max_ind_range,
+                args.data_sub_sample_rate,
+                args.data_randomize,
+                "test",
+                args.raw_data_file,
+                args.processed_data_file,
+                args.memory_map,
+                args.dataset_multiprocessing
+            )
+
+            train_loader = data_loader_terabyte.DataLoader(
+                data_directory=data_directory,
+                data_filename=data_filename,
+                days=list(range(23)),
+                batch_size=args.mini_batch_size,
+                max_ind_range=args.max_ind_range,
+                split="train"
+            )
+
+            test_loader = data_loader_terabyte.DataLoader(
+                data_directory=data_directory,
+                data_filename=data_filename,
+                days=[23],
+                batch_size=args.test_mini_batch_size,
+                max_ind_range=args.max_ind_range,
+                split="test"
+            )
+    else:
+        train_data = CriteoDataset(
+            args.data_set,
+            args.max_ind_range,
+            args.data_sub_sample_rate,
+            args.data_randomize,
+            "train",
+            args.raw_data_file,
+            args.processed_data_file,
+            args.memory_map,
+            args.dataset_multiprocessing,
+        )
+
+        test_data = CriteoDataset(
+            args.data_set,
+            args.max_ind_range,
+            args.data_sub_sample_rate,
+            args.data_randomize,
+            "test",
+            args.raw_data_file,
+            args.processed_data_file,
+            args.memory_map,
+            args.dataset_multiprocessing,
+        )
+
+        collate_wrapper_criteo = collate_wrapper_criteo_offset
+        if offset_to_length_converter:
+            collate_wrapper_criteo = collate_wrapper_criteo_length
+
+        train_loader = torch.utils.data.DataLoader(
+            train_data,
+            batch_size=args.mini_batch_size,
+            shuffle=False,
+            num_workers=args.num_workers,
+            collate_fn=lambda b: collate_wrapper_criteo(b, args),
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+        test_loader = torch.utils.data.DataLoader(
+            test_data,
+            batch_size=args.test_mini_batch_size,
+            shuffle=False,
+            num_workers=args.test_num_workers,
+            collate_fn=lambda b: collate_wrapper_criteo(b, args),
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+    return train_data, train_loader, test_data, test_loader
\ No newline at end of file
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/readme.txt b/models/recommendation/pytorch/torchrec_dlrm/__init__.py
similarity index 100%
rename from benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/readme.txt
rename to models/recommendation/pytorch/torchrec_dlrm/__init__.py
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/readme.txt b/models/recommendation/pytorch/torchrec_dlrm/data_process/__init__.py
similarity index 100%
rename from benchmarks/recommendation/tensorflow/wide_deep_large_ds/readme.txt
rename to models/recommendation/pytorch/torchrec_dlrm/data_process/__init__.py
diff --git a/models/recommendation/pytorch/torchrec_dlrm/data_process/dlrm_dataloader.py b/models/recommendation/pytorch/torchrec_dlrm/data_process/dlrm_dataloader.py
new file mode 100644
index 000000000..994719a29
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/data_process/dlrm_dataloader.py
@@ -0,0 +1,170 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+from typing import List
+
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DAYS,
+    DEFAULT_CAT_NAMES,
+    DEFAULT_INT_NAMES,
+    InMemoryBinaryCriteoIterDataPipe,
+)
+from torchrec.datasets.random import RandomRecDataset
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:multi_hot_criteo
+    from data.multi_hot_criteo import MultiHotCriteoIterDataPipe
+
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot_criteo import MultiHotCriteoIterDataPipe  # noqa F811
+except ImportError:
+    pass
+
+STAGES = ["train", "val", "test"]
+
+
+def _get_random_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    attr = f"limit_{stage}_batches"
+    num_batches = getattr(args, attr)
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    return DataLoader(
+        RandomRecDataset(
+            keys=DEFAULT_CAT_NAMES,
+            batch_size=batch_size,
+            hash_size=args.num_embeddings,
+            hash_sizes=args.num_embeddings_per_feature
+            if hasattr(args, "num_embeddings_per_feature")
+            else None,
+            manual_seed=args.seed if hasattr(args, "seed") else None,
+            ids_per_feature=1,
+            num_dense=len(DEFAULT_INT_NAMES),
+            num_batches=num_batches,
+        ),
+        batch_size=None,
+        batch_sampler=None,
+        pin_memory=args.pin_memory,
+        num_workers=0,
+    )
+
+
+def _get_in_memory_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    if args.in_memory_binary_criteo_path is not None:
+        dir_path = args.in_memory_binary_criteo_path
+        sparse_part = "sparse.npy"
+        datapipe = InMemoryBinaryCriteoIterDataPipe
+    else:
+        dir_path = args.synthetic_multi_hot_criteo_path
+        sparse_part = "sparse_multi_hot.npz"
+        datapipe = MultiHotCriteoIterDataPipe
+
+    if stage == "train":
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS - 1)],
+            [os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS - 1)],
+            [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS - 1)],
+        ]
+    elif stage in ["val", "test"]:
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"day_{DAYS-1}_dense.npy")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_{sparse_part}")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_labels.npy")],
+        ]
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    dataloader = DataLoader(
+        datapipe(
+            stage,
+            *stage_files,  # pyre-ignore[6]
+            batch_size=batch_size,
+            rank=0, #  dist.get_rank(),
+            world_size=1, #  dist.get_world_size(),
+            drop_last=args.drop_last_training_batch if stage == "train" else False,
+            shuffle_batches=args.shuffle_batches,
+            shuffle_training_set=args.shuffle_training_set,
+            shuffle_training_set_random_seed=args.seed,
+            mmap_mode=args.mmap_mode,
+            hashes=args.num_embeddings_per_feature
+            if args.num_embeddings is None
+            else ([args.num_embeddings] * CAT_FEATURE_COUNT),
+        ),
+        batch_size=None,
+        pin_memory=args.pin_memory,
+        collate_fn=lambda x: x,
+    )
+    return dataloader
+
+
+def get_dataloader(args: argparse.Namespace, backend: str, stage: str) -> DataLoader:
+    """
+    Gets desired dataloader from dlrm_main command line options. Currently, this
+    function is able to return either a DataLoader wrapped around a RandomRecDataset or
+    a Dataloader wrapped around an InMemoryBinaryCriteoIterDataPipe.
+
+    Args:
+        args (argparse.Namespace): Command line options supplied to dlrm_main.py's main
+            function.
+        backend (str): "nccl" or "gloo".
+        stage (str): "train", "val", or "test".
+
+    Returns:
+        dataloader (DataLoader): PyTorch dataloader for the specified options.
+
+    """
+    stage = stage.lower()
+    if stage not in STAGES:
+        raise ValueError(f"Supplied stage was {stage}. Must be one of {STAGES}.")
+
+    args.pin_memory = (
+        (backend == "nccl") if not hasattr(args, "pin_memory") else args.pin_memory
+    )
+
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        return _get_random_dataloader(args, stage)
+    else:
+        return _get_in_memory_dataloader(args, stage)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py b/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py
new file mode 100644
index 000000000..9cf15bc51
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py
@@ -0,0 +1,336 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import zipfile
+from typing import Dict, Iterator, List, Optional
+
+import numpy as np
+import torch
+from iopath.common.file_io import PathManager, PathManagerFactory
+from pyre_extensions import none_throws
+from torch.utils.data import IterableDataset
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DEFAULT_CAT_NAMES,
+)
+from torchrec.datasets.utils import Batch, PATH_MANAGER_KEY
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class MultiHotCriteoIterDataPipe(IterableDataset):
+    """
+    Datapipe designed to operate over the MLPerf DLRM v2 synthetic multi-hot dataset.
+    This dataset can be created by following the steps in
+    torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py.
+    Each rank reads only the data for the portion of the dataset it is responsible for.
+
+    Args:
+        stage (str): "train", "val", or "test".
+        dense_paths (List[str]): List of path strings to dense npy files.
+        sparse_paths (List[str]): List of path strings to multi-hot sparse npz files.
+        labels_paths (List[str]): List of path strings to labels npy files.
+        batch_size (int): batch size.
+        rank (int): rank.
+        world_size (int): world size.
+        drop_last (Optional[bool]): Whether to drop the last batch if it is incomplete.
+        shuffle_batches (bool): Whether to shuffle batches
+        shuffle_training_set (bool): Whether to shuffle all samples in the dataset.
+        shuffle_training_set_random_seed (int): The random generator seed used when
+            shuffling the training set.
+        hashes (Optional[int]): List of max categorical feature value for each feature.
+            Length of this list should be CAT_FEATURE_COUNT.
+        path_manager_key (str): Path manager key used to load from different
+            filesystems.
+
+    Example::
+
+        datapipe = MultiHotCriteoIterDataPipe(
+            dense_paths=["day_0_dense.npy"],
+            sparse_paths=["day_0_sparse_multi_hot.npz"],
+            labels_paths=["day_0_labels.npy"],
+            batch_size=1024,
+            rank=torch.distributed.get_rank(),
+            world_size=torch.distributed.get_world_size(),
+        )
+        batch = next(iter(datapipe))
+    """
+
+    def __init__(
+        self,
+        stage: str,
+        dense_paths: List[str],
+        sparse_paths: List[str],
+        labels_paths: List[str],
+        batch_size: int,
+        rank: int,
+        world_size: int,
+        drop_last: Optional[bool] = False,
+        shuffle_batches: bool = False,
+        shuffle_training_set: bool = False,
+        shuffle_training_set_random_seed: int = 0,
+        mmap_mode: bool = False,
+        hashes: Optional[List[int]] = None,
+        path_manager_key: str = PATH_MANAGER_KEY,
+    ) -> None:
+        self.stage = stage
+        self.dense_paths = dense_paths
+        self.sparse_paths = sparse_paths
+        self.labels_paths = labels_paths
+        self.batch_size = batch_size
+        self.rank = rank
+        self.world_size = world_size
+        self.drop_last = drop_last
+        self.shuffle_batches = shuffle_batches
+        self.shuffle_training_set = shuffle_training_set
+        np.random.seed(shuffle_training_set_random_seed)
+        self.mmap_mode = mmap_mode
+        # hashes are not used because they were already applied in the
+        # script that generates the multi-hot dataset.
+        self.hashes: np.ndarray = np.array(hashes).reshape((CAT_FEATURE_COUNT, 1))
+        self.path_manager_key = path_manager_key
+        self.path_manager: PathManager = PathManagerFactory().get(path_manager_key)
+
+        if shuffle_training_set and stage == "train":
+            # Currently not implemented for the materialized multi-hot dataset.
+            self._shuffle_and_load_data_for_rank()
+        else:
+            m = "r" if mmap_mode else None
+            self.dense_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.dense_paths
+            ]
+            self.labels_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.labels_paths
+            ]
+            self.sparse_arrs: List = []
+            for sparse_path in self.sparse_paths:
+                multi_hot_ids_l = []
+                for feat_id_num in range(CAT_FEATURE_COUNT):
+                    multi_hot_ft_ids = self._load_from_npz(
+                        sparse_path, f"{feat_id_num}.npy"
+                    )
+                    multi_hot_ids_l.append(multi_hot_ft_ids)
+                self.sparse_arrs.append(multi_hot_ids_l)
+        len_d0 = len(self.dense_arrs[0])
+        second_half_start_index = int(len_d0 // 2 + len_d0 % 2)
+        if stage == "val":
+            self.dense_arrs[0] = self.dense_arrs[0][:second_half_start_index, :]
+            self.labels_arrs[0] = self.labels_arrs[0][:second_half_start_index, :]
+            self.sparse_arrs[0] = [
+                feats[:second_half_start_index, :] for feats in self.sparse_arrs[0]
+            ]
+        elif stage == "test":
+            self.dense_arrs[0] = self.dense_arrs[0][second_half_start_index:, :]
+            self.labels_arrs[0] = self.labels_arrs[0][second_half_start_index:, :]
+            self.sparse_arrs[0] = [
+                feats[second_half_start_index:, :] for feats in self.sparse_arrs[0]
+            ]
+        # When mmap_mode is enabled, sparse features are hashed when
+        # samples are batched in def __iter__. Otherwise, the dataset has been
+        # preloaded with sparse features hashed in the preload stage, here:
+        # if not self.mmap_mode and self.hashes is not None:
+        #     for k, _ in enumerate(self.sparse_arrs):
+        #         self.sparse_arrs[k] = [
+        #             feat % hash
+        #             for (feat, hash) in zip(self.sparse_arrs[k], self.hashes)
+        #         ]
+
+        self.num_rows_per_file: List[int] = list(map(len, self.dense_arrs))
+        total_rows = sum(self.num_rows_per_file)
+        self.num_full_batches: int = (
+            total_rows // batch_size // self.world_size * self.world_size
+        )
+        self.last_batch_sizes: np.ndarray = np.array(
+            [0 for _ in range(self.world_size)]
+        )
+        remainder = total_rows % (self.world_size * batch_size)
+        if not self.drop_last and 0 < remainder:
+            if remainder < self.world_size:
+                self.num_full_batches -= self.world_size
+                self.last_batch_sizes += batch_size
+            else:
+                self.last_batch_sizes += remainder // self.world_size
+            self.last_batch_sizes[: remainder % self.world_size] += 1
+
+        self.multi_hot_sizes: List[int] = [
+            multi_hot_feat.shape[-1] for multi_hot_feat in self.sparse_arrs[0]
+        ]
+
+        # These values are the same for the KeyedJaggedTensors in all batches, so they
+        # are computed once here. This avoids extra work from the KeyedJaggedTensor sync
+        # functions.
+        self.keys: List[str] = DEFAULT_CAT_NAMES
+        self.index_per_key: Dict[str, int] = {
+            key: i for (i, key) in enumerate(self.keys)
+        }
+
+    def _load_from_npz(self, fname, npy_name):
+        # figure out offset of .npy in .npz
+        zf = zipfile.ZipFile(fname)
+        info = zf.NameToInfo[npy_name]
+        assert info.compress_type == 0
+        zf.fp.seek(info.header_offset + len(info.FileHeader()) + 20)
+        # read .npy header
+        zf.open(npy_name, "r")
+        version = np.lib.format.read_magic(zf.fp)
+        shape, fortran_order, dtype = np.lib.format._read_array_header(zf.fp, version)
+        assert (
+            dtype == "int32"
+        ), f"sparse multi-hot dtype is {dtype} but should be int32"
+        offset = zf.fp.tell()
+        # create memmap
+        return np.memmap(
+            zf.filename,
+            dtype=dtype,
+            shape=shape,
+            order="F" if fortran_order else "C",
+            mode="r",
+            offset=offset,
+        )
+
+    def _np_arrays_to_batch(
+        self,
+        dense: np.ndarray,
+        sparse: List[np.ndarray],
+        labels: np.ndarray,
+    ) -> Batch:
+        if self.shuffle_batches:
+            # Shuffle all 3 in unison
+            shuffler = np.random.permutation(len(dense))
+            sparse = [multi_hot_ft[shuffler, :] for multi_hot_ft in sparse]
+            dense = dense[shuffler]
+            labels = labels[shuffler]
+
+        batch_size = len(dense)
+        lengths = torch.ones((CAT_FEATURE_COUNT * batch_size), dtype=torch.int32)
+        for k, multi_hot_size in enumerate(self.multi_hot_sizes):
+            lengths[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        offsets = torch.cumsum(torch.concat((torch.tensor([0]), lengths)), dim=0)
+        length_per_key = [
+            batch_size * multi_hot_size for multi_hot_size in self.multi_hot_sizes
+        ]
+        offset_per_key = torch.cumsum(
+            torch.concat((torch.tensor([0]), torch.tensor(length_per_key))), dim=0
+        )
+        values = torch.concat([torch.from_numpy(feat.copy()).flatten() for feat in sparse])
+        return Batch(
+            dense_features=torch.from_numpy(dense.copy()),
+            sparse_features=KeyedJaggedTensor(
+                keys=self.keys,
+                values=values,
+                lengths=lengths,
+                offsets=offsets,
+                stride=batch_size,
+                length_per_key=length_per_key,
+                offset_per_key=offset_per_key.tolist(),
+                index_per_key=self.index_per_key,
+            ),
+            labels=torch.from_numpy(labels.reshape(-1).copy()),
+        )
+
+    def __iter__(self) -> Iterator[Batch]:
+        # Invariant: buffer never contains more than batch_size rows.
+        buffer: Optional[List[np.ndarray]] = None
+
+        def append_to_buffer(
+            dense: np.ndarray,
+            sparse: List[np.ndarray],
+            labels: np.ndarray,
+        ) -> None:
+            nonlocal buffer
+            if buffer is None:
+                buffer = [dense, sparse, labels]
+            else:
+                buffer[0] = np.concatenate((buffer[0], dense))
+                buffer[1] = [np.concatenate((b, s)) for b, s in zip(buffer[1], sparse)]
+                buffer[2] = np.concatenate((buffer[2], labels))
+
+        # Maintain a buffer that can contain up to batch_size rows. Fill buffer as
+        # much as possible on each iteration. Only return a new batch when batch_size
+        # rows are filled.
+        file_idx = 0
+        row_idx = 0
+        batch_idx = 0
+        buffer_row_count = 0
+        cur_batch_size = (
+            self.batch_size if self.num_full_batches > 0 else self.last_batch_sizes[0]
+        )
+        while (
+            batch_idx
+            < self.num_full_batches + (self.last_batch_sizes[0] > 0) * self.world_size
+        ):
+            if buffer_row_count == cur_batch_size or file_idx == len(self.dense_arrs):
+                if batch_idx % self.world_size == self.rank:
+                    yield self._np_arrays_to_batch(*none_throws(buffer))
+                    buffer = None
+                buffer_row_count = 0
+                batch_idx += 1
+                if (
+                    0 <= batch_idx - self.num_full_batches < self.world_size
+                    and (self.last_batch_sizes[0] > 0)
+                ):
+                    cur_batch_size = self.last_batch_sizes[
+                        batch_idx - self.num_full_batches
+                    ]
+            else:
+                rows_to_get = min(
+                    cur_batch_size - buffer_row_count,
+                    self.num_rows_per_file[file_idx] - row_idx,
+                )
+                buffer_row_count += rows_to_get
+                slice_ = slice(row_idx, row_idx + rows_to_get)
+
+                if batch_idx % self.world_size == self.rank:
+                    dense_inputs = self.dense_arrs[file_idx][slice_, :]
+                    sparse_inputs = [
+                        feats[slice_, :] for feats in self.sparse_arrs[file_idx]
+                    ]
+                    target_labels = self.labels_arrs[file_idx][slice_, :]
+
+                    # if self.mmap_mode and self.hashes is not None:
+                    #     sparse_inputs = [
+                    #         feats % hash
+                    #         for (feats, hash) in zip(sparse_inputs, self.hashes)
+                    #     ]
+
+                    append_to_buffer(
+                        dense_inputs,
+                        sparse_inputs,
+                        target_labels,
+                    )
+                row_idx += rows_to_get
+
+                if row_idx >= self.num_rows_per_file[file_idx]:
+                    file_idx += 1
+                    row_idx = 0
+
+    def __len__(self) -> int:
+        return self.num_full_batches // self.world_size + (self.last_batch_sizes[0] > 0)
+
+    def load_batch(self, sample_list=None) -> Batch:
+        if sample_list is None:
+            sample_list = list(range(self.batch_size))
+        dense = self.dense_arrs[0][sample_list, :]
+        sparse = [arr[sample_list, :] for arr in self.sparse_arrs[0]]
+        labels = self.labels_arrs[0][sample_list, :]
+        return self._np_arrays_to_batch(dense, sparse, labels)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py b/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
new file mode 100644
index 000000000..6de291bf7
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
@@ -0,0 +1,1257 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import argparse
+import itertools
+import os
+import sys
+from torch.profiler import record_function, ProfilerActivity
+from dataclasses import dataclass, field
+from enum import Enum
+from pprint import pprint
+from typing import List, Optional
+import time
+
+import torch
+print(f"torch version {torch.__version__}")
+import torchmetrics as metrics
+from pyre_extensions import none_throws
+from torch.utils.data import DataLoader
+from torchrec import EmbeddingBagCollection
+from torchrec.datasets.criteo import DEFAULT_CAT_NAMES, DEFAULT_INT_NAMES
+
+from torchrec.models.dlrm import DLRM, DLRM_DCN, DLRM_Projection, DLRMTrain
+from torchrec.modules.embedding_configs import EmbeddingBagConfig
+from torchrec.optim.apply_optimizer_in_backward import apply_optimizer_in_backward
+from torchrec.optim.keyed import CombinedOptimizer, KeyedOptimizerWrapper
+from torchrec.optim.optimizers import in_backward_optimizer_filter
+from tqdm import tqdm
+from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
+import intel_extension_for_pytorch as ipex
+print(f"IPEX version {ipex.__version__}")
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:dlrm_dataloader
+    from data_process.dlrm_dataloader import get_dataloader
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:lr_scheduler
+    from lr_scheduler import LRPolicyScheduler
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot, RestartableMap
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .data_process.dlrm_dataloader import get_dataloader  # noqa F811
+    from .lr_scheduler import LRPolicyScheduler  # noqa F811
+    from .multi_hot import Multihot, RestartableMap  # noqa F811
+except ImportError:
+    pass
+
+TRAIN_PIPELINE_STAGES = 3  # Number of stages in TrainPipelineSparseDist.
+# Optimizer parameters:
+ADAGRAD_LR_DECAY = 0
+ADAGRAD_INIT_ACC = 0
+ADAGRAD_EPS = 1e-8
+WEIGHT_DECAY = 0
+
+import logging
+logger: logging.Logger = logging.getLogger(__name__)
+logger.setLevel(1)
+
+class InteractionType(Enum):
+    ORIGINAL = "original"
+    DCN = "dcn"
+    PROJECTION = "projection"
+
+    def __str__(self):
+        return self.value
+
+def load_snapshot(model, snapshot_dir):
+    from torchsnapshot import Snapshot
+    snapshot = Snapshot(path=snapshot_dir)
+    snapshot.restore(app_state={"model": model})
+
+def trace_handler(prof):
+    print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+    prof.export_chrome_trace(f"{prof._mode}-{prof._dtype}-dlrm_trace_step{prof.step_num}.json")
+
+prof_schedule=torch.profiler.schedule(
+    wait=10,
+    warmup=10,
+    active=1,
+    repeat=10
+)
+
+def print_memory(stage):
+    import os
+    import psutil
+    print("dlrmv2-memory-usage-log: ", time.time(), stage, psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024)
+
+def fetch_batch(dataloader):
+    try:
+        batch = dataloader.dataset.load_batch()
+    except:
+        batch = dataloader.source.dataset.batch_generator._generate_batch()
+    return batch
+
+def parse_autocast(dtype: str):
+    _dtype = None
+    autocast = False
+    if dtype == 'bf16':
+        autocast = True
+        _dtype = torch.bfloat16
+    elif dtype == 'fp16':
+        autocast= True
+        _dtype = torch.float16
+    elif dtype == 'int8':
+        _dtype = torch.int8
+    else:
+        assert dtype in ['fp32', 'bf32']
+        _dtype = torch.float
+    return autocast, _dtype
+
+def convert_int8(args, model, dataloader):
+    from torch.ao.quantization import HistogramObserver, PerChannelMinMaxObserver, QConfig
+    from intel_extension_for_pytorch.quantization import prepare, convert
+    qconfig = QConfig(
+        activation=HistogramObserver.with_args(qscheme=torch.per_tensor_symmetric, dtype=torch.qint8, bins=127, upsample_rate=256, quant_min= -127, quant_max=126),
+        weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric)
+    )
+    batch = fetch_batch(dataloader)
+    batch.sparse_features = unpack(batch.sparse_features)
+    model = prepare(
+        model,
+        qconfig,
+        example_inputs=(batch.dense_features, batch.sparse_features),
+        inplace=True
+    )
+    if args.calibration:
+        # https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v2/pytorch#calibration-set
+        assert args.synthetic_multi_hot_criteo_path, "need real dataset to calibrate"
+        batch_idx = list(range(128000))
+        batch = dataloader.dataset.load_batch(batch_idx)
+        batch.sparse_features = unpack(batch.sparse_features)
+        model(batch.dense_features, batch.sparse_features)
+        model.save_qconf_summary(qconf_summary=args.int8_configure_dir)
+        logger.info("calibration done and save to %s", args.int8_configure_dir)
+        exit()
+    else:
+        model.load_qconf_summary(qconf_summary = args.int8_configure_dir)
+        convert(model, inplace=True)
+        model.eval()
+        torch._C._jit_set_texpr_fuser_enabled(False)
+        model = torch.jit.trace(model, (batch.dense_features, batch.sparse_features), check_trace=True)
+        model = torch.jit.freeze(model)
+        model(batch.dense_features, batch.sparse_features)
+        model(batch.dense_features, batch.sparse_features)
+        print(model.graph_for(batch.dense_features, batch.sparse_features))
+        return model
+
+def ipex_optimize(args, model, optimizer, dataloader):
+    example_batch = fetch_batch(dataloader)
+    example_batch.sparse_features = unpack(example_batch.sparse_features)
+    dense, sparse = example_batch.dense_features, example_batch.sparse_features
+    autocast, dtype = parse_autocast(args.dtype)
+    auto_kernel_selection = True if args.dtype == 'bf32' else False
+    import intel_extension_for_pytorch as ipex
+    if dtype == torch.int8:
+        assert args.inference_only
+        with torch.no_grad():
+            model = convert_int8(args, model, dataloader)
+    elif args.inference_only:
+        with torch.no_grad():
+            model = ipex.optimize(
+                model=model.eval(),
+                dtype=dtype,
+                sample_input=(dense, sparse),
+                inplace=True,
+                auto_kernel_selection=auto_kernel_selection,
+                linear_bn_folding=False,
+                conv_bn_folding=False
+            )
+            if args.dtype == 'bf32':
+                ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+            if args.jit:
+                with torch.cpu.amp.autocast(enabled=autocast, dtype=dtype):
+                    torch._C._jit_set_texpr_fuser_enabled(False)
+                    model = torch.jit.trace(model, (dense, sparse))
+                    model = torch.jit.freeze(model)
+                    model(dense, sparse)
+                    model(dense, sparse)
+    else:
+        model, optimizer = ipex.optimize(
+            model=model,
+            optimizer=optimizer,
+            dtype=dtype,
+            sample_input=(dense, sparse),
+            inplace=True,
+            auto_kernel_selection=auto_kernel_selection,
+            linear_bn_folding=False,
+            conv_bn_folding=False
+        )
+        pass
+    if args.dtype == 'bf32':
+        ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+    return model, optimizer
+
+def parse_args(argv: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
+    parser.add_argument(
+        "--epochs",
+        type=int,
+        default=1,
+        help="number of epochs to train",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=32,
+        help="batch size to use for training",
+    )
+    parser.add_argument(
+        "--drop_last_training_batch",
+        dest="drop_last_training_batch",
+        action="store_true",
+        help="Drop the last non-full training batch",
+    )
+    parser.add_argument(
+        "--test_batch_size",
+        type=int,
+        default=None,
+        help="batch size to use for validation and testing",
+    )
+    parser.add_argument(
+        "--limit_train_batches",
+        type=int,
+        default=None,
+        help="number of train batches",
+    )
+    parser.add_argument(
+        "--limit_val_batches",
+        type=int,
+        default=None,
+        help="number of validation batches",
+    )
+    parser.add_argument(
+        "--limit_test_batches",
+        type=int,
+        default=None,
+        help="number of test batches",
+    )
+    parser.add_argument(
+        "--dataset_name",
+        type=str,
+        default="criteo_1t",
+        help="dataset for experiment, current support criteo_1tb, criteo_kaggle",
+    )
+    parser.add_argument(
+        "--num_embeddings",
+        type=int,
+        default=100_000,
+        help="max_ind_size. The number of embeddings in each embedding table. Defaults"
+        " to 100_000 if num_embeddings_per_feature is not supplied.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        default=None,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--dense_arch_layer_sizes",
+        type=str,
+        default="512,256,64",
+        help="Comma separated layer sizes for dense arch.",
+    )
+    parser.add_argument(
+        "--over_arch_layer_sizes",
+        type=str,
+        default="512,512,256,1",
+        help="Comma separated layer sizes for over arch.",
+    )
+    parser.add_argument(
+        "--embedding_dim",
+        type=int,
+        default=64,
+        help="Size of each embedding.",
+    )
+    parser.add_argument(
+        "--interaction_branch1_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch1 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--interaction_branch2_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch2 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--dcn_num_layers",
+        type=int,
+        default=3,
+        help="Number of DCN layers in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--dcn_low_rank_dim",
+        type=int,
+        default=512,
+        help="Low rank dimension for DCN in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=None,
+        help="Random seed for reproducibility.",
+    )
+    parser.add_argument(
+        "--pin_memory",
+        dest="pin_memory",
+        action="store_true",
+        help="Use pinned memory when loading data.",
+    )
+    parser.add_argument(
+        "--mmap_mode",
+        dest="mmap_mode",
+        action="store_true",
+        help="--mmap_mode mmaps the dataset."
+        " That is, the dataset is kept on disk but is accessed as if it were in memory."
+        " --mmap_mode is intended mostly for faster debugging. Use --mmap_mode to bypass"
+        " preloading the dataset when preloading takes too long or when there is "
+        " insufficient memory available to load the full dataset.",
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the Criteo dataset npy files.",
+    )
+    parser.add_argument(
+        "--synthetic_multi_hot_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the MLPerf v2 synthetic multi-hot dataset npz files.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=15.0,
+        help="Learning rate.",
+    )
+    parser.add_argument(
+        "--shuffle_batches",
+        dest="shuffle_batches",
+        action="store_true",
+        help="Shuffle each batch during training.",
+    )
+    parser.add_argument(
+        "--shuffle_training_set",
+        dest="shuffle_training_set",
+        action="store_true",
+        help="Shuffle the training set in memory. This will override mmap_mode",
+    )
+    parser.add_argument(
+        "--validation_freq_within_epoch",
+        type=int,
+        default=None,
+        help="Frequency at which validation will be run within an epoch.",
+    )
+    parser.add_argument(
+        "--validation_auroc",
+        type=float,
+        default=None,
+        help="Validation AUROC threshold to stop training once reached.",
+    )
+    parser.add_argument(
+        "--evaluate_on_epoch_end",
+        action="store_true",
+        help="Evaluate using validation set on each epoch end.",
+    )
+    parser.add_argument(
+        "--evaluate_on_training_end",
+        action="store_true",
+        help="Evaluate using test set on training end.",
+    )
+    parser.add_argument(
+        "--adagrad",
+        dest="adagrad",
+        action="store_true",
+        help="Flag to determine if adagrad optimizer should be used.",
+    )
+    parser.add_argument(
+        "--interaction_type",
+        type=InteractionType,
+        choices=list(InteractionType),
+        default=InteractionType.ORIGINAL,
+        help="Determine the interaction type to be used (original, dcn, or projection)"
+        " default is original DLRM with pairwise dot product",
+    )
+    parser.add_argument(
+        "--collect_multi_hot_freqs_stats",
+        dest="collect_multi_hot_freqs_stats",
+        action="store_true",
+        help="Flag to determine whether to collect stats on freq of embedding access.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        default=None,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default=None,
+        help="Multi-hot distribution options.",
+    )
+    parser.add_argument(
+        "--lr_warmup_steps",
+        type=int,
+        default=0,
+    )
+    parser.add_argument(
+        "--lr_decay_start",
+        type=int,
+        default=0,
+    )
+    parser.add_argument(
+        "--lr_decay_steps",
+        type=int,
+        default=0,
+    )
+    parser.add_argument(
+        "--print_lr",
+        action="store_true",
+        help="Print learning rate every iteration.",
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help="Enable TensorFloat-32 mode for matrix multiplications on A100 (or newer) GPUs.",
+    )
+    parser.add_argument(
+        "--print_sharding_plan",
+        action="store_true",
+        help="Print the sharding plan used for each embedding table.",
+    )
+    parser.add_argument(
+        "--print_progress",
+        action="store_true",
+        help="Print tqdm progress bar during training and evaluation.",
+    )
+    parser.add_argument(
+        "--ipex-optimize",
+        action="store_true",
+        help="Whether optimize model by ipex.optimize",
+    )
+    parser.add_argument(
+        "--dtype",
+        type=str,
+        choices=["fp32", "bf16", "fp16", "int8", "bf32"],
+        default=None,
+        help="Model dtypes.",
+    )
+    parser.add_argument(
+        "--inference-only",
+        action="store_true",
+        help="Whether only run inference path.",
+    )
+    parser.add_argument(
+        "--int8-configure-dir",
+        type=str,
+        default="./int8_configure.json",
+        help="Int8 recipe location.",
+    )
+    parser.add_argument(
+        "--calibration",
+        action="store_true",
+        help="Whether calibration only for this run.",
+    )
+    parser.add_argument(
+        "--test_auroc",
+        action="store_true",
+        help="Test auroc.",
+    )    
+    parser.add_argument(
+        "--log-freq",
+        type=int,
+        default=0,
+        help="log-freq to print performance statistics.",
+    )
+    parser.add_argument(
+        "--snapshot-dir",
+        type=str,
+    )
+    parser.add_argument(
+        "--profile",
+        action="store_true",
+        help="Test auroc.",
+    )
+    parser.add_argument(
+        "--jit",
+        action="store_true",
+        help="Test auroc.",
+    )
+    parser.add_argument(
+        "--benchmark",
+        action="store_true",
+        help="use same samples to benchmark",
+    )
+    parser.add_argument(
+        "--ipex-merged-emb-cat",
+        action="store_true",
+        help="whether use ipex customer op for merged_emb_cat",
+    )
+    parser.add_argument(
+        "--share-weight-instance",
+        type=int,
+        default=0,
+        help="if value > 0, will use pytorch throughputbenchmark to share weight with `value` instance",
+    )
+    return parser.parse_args(argv)
+
+
+def _evaluate(
+    eval_model,
+    eval_dataloader: DataLoader,
+    stage: str,
+    epoch_num: float,
+    args,
+) -> float:
+    """
+    Evaluates model. Computes and prints AUROC. Helper function for train_val_test.
+
+    Args:
+        limit_batches (Optional[int]): Limits the dataloader to the first `limit_batches` batches.
+        eval_pipeline (TrainPipelineSparseDist): pipelined model.
+        eval_dataloader (DataLoader): Dataloader for either the validation set or test set.
+        stage (str): "val" or "test".
+        epoch_num (float): Iterations passed as epoch fraction (for logging purposes).
+        print_progress (bool): Whether to print tqdm progress bar.
+
+    Returns:
+        float: auroc result
+    """
+    limit_batches = args.limit_val_batches
+    print_progress = args.print_progress
+    autocast_enabled, autocast_dtype = parse_autocast(args.dtype)
+    log_freq = args.log_freq
+    enable_torch_profile = args.profile
+
+    benckmark_batch = fetch_batch(eval_dataloader)
+    benckmark_batch.sparse_features = unpack(benckmark_batch.sparse_features)
+
+    def fetch_next(iterator, current_it):
+        if args.benchmark:
+            if current_it == limit_batches:
+                raise StopIteration
+            return benckmark_batch
+        else:
+            with record_function("generate batch"):
+                next_batch = next(iterator)
+            with record_function("unpack KeyJaggedTensor"):
+                next_batch.sparse_features = unpack(next_batch.sparse_features)
+            return next_batch
+
+    def eval_step(model, iterator, current_it):
+        batch = fetch_next(iterator, current_it)
+        with record_function("model forward"):
+            t1 = time.time()
+            logits = model(batch.dense_features, batch.sparse_features)
+            t2 = time.time()
+        return logits, batch.labels, t2 - t1
+
+        
+    pbar = tqdm(
+        iter(int, 1),
+        desc=f"Evaluating {stage} set",
+        total=len(eval_dataloader),
+        disable=not print_progress,
+    )
+
+    logger.info(f"EVAL_START, EPOCH_NUM: {epoch_num}")
+
+    eval_model.eval()
+    device = torch.device('cpu')
+
+    iterator = itertools.islice(iter(eval_dataloader), limit_batches)
+    # Two filler batches are appended to the end of the iterator to keep the pipeline active while the
+    # last two remaining batches are still in progress awaiting results.
+    two_filler_batches = itertools.islice(
+        iter(eval_dataloader), TRAIN_PIPELINE_STAGES - 1
+    )
+    iterator = itertools.chain(iterator, two_filler_batches)
+
+    preds = []
+    labels = []
+    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+    total_t = 0
+    it = 1
+    ctx1 = torch.no_grad()
+    ctx2 = torch.cpu.amp.autocast(enabled=autocast_enabled, dtype=autocast_dtype)
+    ctx3 = torch.profiler.profile(activities=[ProfilerActivity.CPU], schedule=prof_schedule, on_trace_ready=trace_handler)
+    with ctx1, ctx2:
+        if enable_torch_profile:
+            p = ctx3.__enter__()
+            setattr(p, '_dtype', autocast_dtype)
+            setattr(p, '_mode', 'eval')
+        while True:
+            try:
+                logits, label, fw_t = eval_step(eval_model, iterator, it)
+                total_t += fw_t
+                if log_freq != 0 and it % log_freq == 0:
+                    preds = [torch.cat(preds)]
+                    labels = [torch.cat(labels)]
+                    num_samples = labels[0].shape[0]
+                    auroc = ipex._C.roc_auc_score(labels[0].float(), preds[0].squeeze().float())
+                    logger.info(f"avg eval time per iter at ITER: {it}, {total_t/it} s, num_samples: {num_samples}, AUROC: {auroc}")
+                pred = torch.sigmoid(logits)
+                preds.append(pred)
+                labels.append(label)
+                pbar.update(1)
+                it += 1
+                if enable_torch_profile:
+                    p.step()
+            except StopIteration:
+                # Dataset traversal complete
+                break
+        if enable_torch_profile:
+            ctx3.__exit__(None, None, None)
+
+    preds = torch.cat(preds)
+    labels = torch.cat(labels)
+    num_samples = labels.shape[0]
+    auroc = ipex._C.roc_auc_score(labels.float(), preds.squeeze().float())
+
+    logger.info(f"AUROC over {stage} set: {auroc}.")
+    logger.info(f"Number of {stage} samples: {num_samples}")
+    logger.info(f"Throughput: {num_samples/total_t} fps")
+    logger.info(f"Final AUROC: {auroc} ")
+
+    return auroc
+
+
+def _train(
+    train_model,
+    train_optimizer,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    epoch: int,
+    lr_scheduler,
+    args,
+) -> bool:
+    """
+    Trains model for 1 epoch. Helper function for train_val_test.
+
+    Args:
+        train_pipeline (TrainPipelineSparseDist): pipelined model used for training.
+        val_pipeline (TrainPipelineSparseDist): pipelined model used for validation.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        epoch (int): The number of complete passes through the training set so far.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+        print_lr (bool): Whether to print the learning rate every training step.
+        validation_freq (Optional[int]): The number of training steps between validation runs within an epoch.
+        validation_auroc (Optional[float]): AUROC level desired for stopping training.
+        limit_train_batches (Optional[int]): Limits the training set to the first `limit_train_batches` batches.
+        limit_val_batches (Optional[int]): Limits the validation set to the first `limit_val_batches` batches.
+        print_progress (bool): Whether to print tqdm progress bar.
+
+    Returns:
+        bool: Whether the validation_auroc threshold is reached.
+    """
+    print_lr = args.print_lr
+    validation_freq = args.validation_freq_within_epoch
+    validation_auroc = args.validation_auroc
+    limit_train_batches = args.limit_train_batches
+    print_progress = args.print_progress
+    autocast_enabled, autocast_dtype = parse_autocast(args.dtype)
+    log_freq = args.log_freq
+    enable_torch_profile = args.profile
+
+    train_model.train()
+    if autocast_dtype == torch.float16:
+        scaler = torch.cpu.amp.GradScaler()
+
+    benckmark_batch = fetch_batch(train_dataloader)
+    benckmark_batch.sparse_features = unpack(benckmark_batch.sparse_features)
+
+    def fetch_next(iterator, current_it):
+        if args.benchmark:
+            if current_it == limit_train_batches:
+                raise StopIteration
+            return benckmark_batch
+        else:
+            with record_function("generate batch"):
+                next_batch = next(iterator)
+            with record_function("unpack KeyJaggedTensor"):
+                next_batch.sparse_features = unpack(next_batch.sparse_features)
+            return next_batch
+
+    def train_step(model, opt, iterator, current_it):
+        next_batch = fetch_next(iterator, current_it)
+        t1 = time.time()
+        with record_function("zero_grad"):
+            opt.zero_grad(set_to_none=True)
+        with record_function("fw"):
+            if autocast_enabled:
+                with torch.cpu.amp.autocast(enabled=autocast_enabled, dtype=autocast_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+            else:
+                losses, _ = model(next_batch)
+                loss = torch.sum(losses, dim=0)
+        with record_function("bw"):
+            if autocast_dtype == torch.float16:
+                scaler.scale(loss).backward()
+            else:
+                loss.backward()
+        with record_function("optimizer update"):
+            if autocast_dtype == torch.float16:
+                scaler.step(opt)
+                scaler.update()
+            else:
+                opt.step()
+                pass
+        train_t = time.time() - t1
+        return next_batch.dense_features.shape[0], train_t
+
+    iterator = itertools.islice(iter(train_dataloader), limit_train_batches)
+    # Two filler batches are appended to the end of the iterator to keep the pipeline active while the
+    # last two remaining batches are still in progress awaiting results.
+    two_filler_batches = itertools.islice(
+        iter(train_dataloader), TRAIN_PIPELINE_STAGES - 1
+    )
+    iterator = itertools.chain(iterator, two_filler_batches)
+
+    pbar = tqdm(
+        iter(int, 1),
+        desc=f"Epoch {epoch}",
+        total=len(train_dataloader),
+        disable=not print_progress,
+    )
+
+    it = 1
+    is_success = False
+    total_t = 0
+    num_samples = 0
+    ctx = torch.profiler.profile(activities=[ProfilerActivity.CPU], schedule=prof_schedule, on_trace_ready=trace_handler)
+    if enable_torch_profile:
+        p = ctx.__enter__()
+        setattr(p, '_dtype', autocast_dtype)
+        setattr(p, '_mode', 'train')
+    for it in itertools.count(1):
+        try:
+            if  print_lr:
+                for i, g in enumerate(train_optimizer.param_groups):
+                    logger.info(f"lr: {it} {i} {g['lr']:.6f}")
+            samples, train_t = train_step(train_model, train_optimizer, iterator, it)
+            if enable_torch_profile:
+                p.step()
+            num_samples += samples
+            total_t += train_t
+
+            if log_freq != 0 and it % log_freq == 0:
+                logger.info(f"avg training time per iter at ITER: {it}, {total_t/it} s")
+                print_memory(f"memory usage at iter {it}")
+
+            lr_scheduler.step()
+            pbar.update(1)
+            if validation_freq and it % validation_freq == 0:
+                epoch_num = epoch + it / len(train_dataloader)
+                auroc_result = _evaluate(
+                    train_model.model,
+                    val_dataloader,
+                    "val",
+                    epoch_num,
+                    args
+                )
+                if validation_auroc is not None and auroc_result >= validation_auroc:
+                    logger.info("auc = {auroc_result} >= validation_auroc {validation_auroc}, stopped since success")
+                    is_success = True
+                    break
+                train_model.train()
+        except StopIteration:
+            # Dataset traversal complete
+            break
+    if enable_torch_profile:
+        ctx.__exit__(None, None, None)
+
+    logger.info(f"Total number of iterations: {it - 1}")
+    logger.info(f"Throughput: {num_samples/total_t} fps")
+
+    return is_success
+
+def _share_weight_benchmark(
+    model,
+    data_loader,
+    args,
+):
+    from torch.utils import ThroughputBenchmark
+    print_memory("start to init throughput benchmark")
+    bench = ThroughputBenchmark(model)
+    batch = fetch_batch(data_loader)
+    batch.sparse_features = unpack(batch.sparse_features)
+    print_memory("start to add input to throughput benchmark")
+    bench.add_input(batch.dense_features, batch.sparse_features)
+    print_memory("start to run throughput benchmark")
+    stats = bench.benchmark(
+        num_calling_threads=args.share_weight_instance,
+        num_warmup_iters=100,
+        num_iters=args.limit_val_batches * args.share_weight_instance,
+    )
+    print(stats)
+    latency = stats.latency_avg_ms
+    batch_size = batch.dense_features.shape[0]
+    throughput = (1 / latency) * 1000 * batch_size * args.share_weight_instance
+    print("Throughput: {:.3f} fps".format(throughput))
+
+@dataclass
+class TrainValTestResults:
+    val_aurocs: List[float] = field(default_factory=list)
+    test_auroc: Optional[float] = None
+
+
+def train_val_test(
+    args: argparse.Namespace,
+    model: torch.nn.Module,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    test_dataloader: DataLoader,
+    lr_scheduler: LRPolicyScheduler,
+) -> TrainValTestResults:
+    """
+    Train/validation/test loop.
+
+    Args:
+        args (argparse.Namespace): parsed command line args.
+        model (torch.nn.Module): model to train.
+        optimizer (torch.optim.Optimizer): optimizer to use.
+        device (torch.device): device to use.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        test_dataloader (DataLoader): Test set's dataloader.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+
+    Returns:
+        TrainValTestResults.
+    """
+    results = TrainValTestResults()
+
+    if args.inference_only:
+        if args.share_weight_instance > 0:
+            _share_weight_benchmark(
+                model.model,
+                val_dataloader,
+                args
+            )
+            exit()
+        # Mlperf is using val set to test auroc
+        # https://github.com/mlcommons/inference/blob/master/recommendation/dlrm_v2/pytorch/python/multihot_criteo.py#L99-L107
+        test_auroc = _evaluate(
+            model.model,
+            val_dataloader,
+            "test",
+            0,
+            args,
+        )
+        results.test_auroc = test_auroc
+        return results
+
+    epoch = 0
+    is_success = False
+    for epoch in range(args.epochs):
+        logger.info(f"EPOCH_START, EPOCH_NUM: {epoch}")
+        is_success = _train(
+            model,
+            optimizer,
+            train_dataloader,
+            val_dataloader,
+            epoch,
+            lr_scheduler,
+            args,
+        )
+        if args.evaluate_on_epoch_end:
+            val_auroc = _evaluate(
+                model.model,
+                val_dataloader,
+                "val",
+                epoch + 1,
+                args,
+            )
+            results.val_aurocs.append(val_auroc)
+        # logger.end(
+        #     key=mllog_constants.EPOCH_STOP,
+        #     metadata={mllog_constants.EPOCH_NUM: epoch},
+        # )
+        if is_success:
+            break
+
+    if is_success:
+        pass
+        # Run status "aborted" is reported in the case AUROC threshold is not met
+        # logger.end(
+        #     key=mllog_constants.RUN_STOP,
+        #     metadata={
+        #         mllog_constants.STATUS: mllog_constants.ABORTED,
+        #         mllog_constants.EPOCH_NUM: epoch + 1,
+        #     },
+        # )
+
+    if args.evaluate_on_training_end:
+        test_auroc = _evaluate(
+            model.model,
+            test_dataloader,
+            "test",
+            epoch + 1,
+            args,
+        )
+        results.test_auroc = test_auroc
+
+    return results
+
+
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+
+    args = parse_args(argv)
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+
+    if args.multi_hot_sizes is not None:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    device: torch.device = torch.device("cpu")
+    backend = "gloo"
+
+    pprint(vars(args))
+    # logger.event(
+    #     key=mllog_constants.GLOBAL_BATCH_SIZE,
+    #     value=dist.get_world_size() * args.batch_size,
+    # )
+    # logger.event(
+    #     key=mllog_constants.GRADIENT_ACCUMULATION_STEPS,
+    #     value=1,  # Gradient accumulation is not supported in the reference implementation
+    # )
+    # logger.event(
+    #     key=mllog_constants.SEED,
+    #     value=args.seed,  # Seeding model is not supported in the reference implementation
+    # )
+
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    eb_configs = [
+        EmbeddingBagConfig(
+            name=f"t_{feature_name}",
+            embedding_dim=args.embedding_dim,
+            num_embeddings=none_throws(args.num_embeddings_per_feature)[feature_idx]
+            if args.num_embeddings is None
+            else args.num_embeddings,
+            feature_names=[feature_name],
+        )
+        for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
+    ]
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    print_memory("start init dlrm ")
+    if args.interaction_type == InteractionType.ORIGINAL:
+        dlrm_model = DLRM(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.DCN:
+        if args.ipex_optimize:
+            from ipex_optimized_model.model import IPEX_DLRM_DCN
+            dcn_init_fn = IPEX_DLRM_DCN
+        else:
+            dcn_init_fn = DLRM_DCN
+        dlrm_model = dcn_init_fn(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("cpu")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dcn_num_layers=args.dcn_num_layers,
+            dcn_low_rank_dim=args.dcn_low_rank_dim,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.PROJECTION:
+        dlrm_model = DLRM_Projection(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
+            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
+            dense_device=device,
+        )
+    else:
+        raise ValueError(
+            "Unknown interaction option set. Should be original, dcn, or projection."
+        )
+
+    train_model = DLRMTrain(dlrm_model)
+    if args.test_auroc or args.calibration:
+        assert args.snapshot_dir
+        print_memory("start loading checkpoint ")
+        load_snapshot(train_model, args.snapshot_dir)
+    
+    from ipex_optimized_model.model import replace_crossnet, replace_embeddingbag_collection
+    replace_crossnet(train_model.model)
+    if args.inference_only and args.ipex_merged_emb_cat:
+        replace_embeddingbag_collection(train_model.model)
+    else:
+        for emb in train_model.model.sparse_arch.embedding_bag_collection.embedding_bags.values():
+            emb.sparse = True
+    # embedding_optimizer = torch.optim.Adagrad if args.adagrad else torch.optim.SGD
+    # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
+    # the optimizer update will be applied in the backward pass, in this case through a fused op.
+    # TorchRec will use the FBGEMM implementation of EXACT_ADAGRAD. For GPU devices, a fused CUDA kernel is invoked. For CPU, FBGEMM_GPU invokes CPU kernels
+    # https://github.com/pytorch/FBGEMM/blob/2cb8b0dff3e67f9a009c4299defbd6b99cc12b8f/fbgemm_gpu/fbgemm_gpu/split_table_batched_embeddings_ops.py#L676-L678
+
+    # Note that lr_decay, weight_decay and initial_accumulator_value for Adagrad optimizer in FBGEMM v0.3.2
+    # cannot be specified below. This equivalently means that all these parameters are hardcoded to zero.
+    optimizer_kwargs = {"lr": args.learning_rate}
+    if args.adagrad:
+        optimizer_kwargs["eps"] = ADAGRAD_EPS
+    # apply_optimizer_in_backward(
+    #     embedding_optimizer,
+    #     train_model.model.sparse_arch.parameters(),
+    #     optimizer_kwargs,
+    # )
+    model = train_model
+    # model.model.sparse_arch = SparseArchTraceAbleWrapper(model.model.sparse_arch)
+
+    # def optimizer_with_params():
+    #     if args.adagrad:
+    #         return lambda params: torch.optim.Adagrad(
+    #             params,
+    #             lr=args.learning_rate,
+    #             lr_decay=ADAGRAD_LR_DECAY,
+    #             weight_decay=WEIGHT_DECAY,
+    #             initial_accumulator_value=ADAGRAD_INIT_ACC,
+    #             eps=ADAGRAD_EPS,
+    #         )
+    #     else:
+    #         return lambda params: torch.optim.SGD(
+    #             params,
+    #             lr=args.learning_rate,
+    #             weight_decay=WEIGHT_DECAY,
+    #         )
+
+    # dense_optimizer = KeyedOptimizerWrapper(
+    #     dict(in_backward_optimizer_filter(model.named_parameters())),
+    #     optimizer_with_params(),
+    # )
+    # optimizer = CombinedOptimizer([model.fused_optimizer, dense_optimizer])
+    optimizer,  lr_scheduler = None, None
+    if not args.inference_only:
+        print_memory("start create optimizer ")
+        if args.adagrad:
+            optimizer = torch.optim.Adagrad(
+                model.parameters(),
+                lr=args.learning_rate,
+                lr_decay=ADAGRAD_LR_DECAY,
+                weight_decay=WEIGHT_DECAY,
+                initial_accumulator_value=ADAGRAD_INIT_ACC,
+                eps=ADAGRAD_EPS,
+            )
+        else:
+            optimizer = torch.optim.SGD(
+                model.parameters(),
+                lr=args.learning_rate,
+                weight_decay=WEIGHT_DECAY,
+            )
+
+        lr_scheduler = LRPolicyScheduler(
+            optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
+        )
+
+    # logger.event(
+    #     key=mllog_constants.OPT_NAME,
+    #     value=mllog_constants.ADAGRAD if args.adagrad else mllog_constants.SGD,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_BASE_LR,
+    #     value=args.learning_rate,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_ADAGRAD_LR_DECAY,
+    #     value=ADAGRAD_LR_DECAY,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_WEIGHT_DECAY,
+    #     value=WEIGHT_DECAY,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_ADAGRAD_INITIAL_ACCUMULATOR_VALUE,
+    #     value=ADAGRAD_INIT_ACC,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_ADAGRAD_EPSILON,
+    #     value=ADAGRAD_EPS,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_LR_WARMUP_STEPS,
+    #     value=args.lr_warmup_steps,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_LR_DECAY_START_STEP,
+    #     value=args.lr_decay_start,
+    # )
+    # logger.event(
+    #     key=mllog_constants.OPT_LR_DECAY_STEPS,
+    #     value=args.lr_decay_steps,
+    # )
+
+    print_memory("start create dataloader ")
+
+    # only create 1 multi-hot sample to save memory
+    if args.multi_hot_sizes is not None:
+        vb, tb = args.limit_val_batches, args.limit_test_batches
+        args.limit_val_batches = 1
+        args.limit_test_batches = 1
+
+    val_dataloader = get_dataloader(args, backend, "val")
+    test_dataloader = get_dataloader(args, backend, "test")
+
+    if args.multi_hot_sizes is not None:
+        args.limit_val_batches, args.limit_test_batches = vb, tb 
+
+    if args.inference_only:
+        train_dataloader = None
+    else:
+        train_dataloader = get_dataloader(args, backend, "train")
+
+    # logger.event(
+    #     key=mllog_constants.TRAIN_SAMPLES,
+    #     value=dist.get_world_size() * len(train_dataloader) * args.batch_size,
+    # )
+
+    if args.multi_hot_sizes is not None:
+        print_memory("start to create Multihot for dummy data ")
+        multihot = Multihot(
+            args.multi_hot_sizes,
+            args.num_embeddings_per_feature,
+            args.batch_size,
+            collect_freqs_stats=args.collect_multi_hot_freqs_stats,
+            dist_type=args.multi_hot_distribution_type,
+        )
+        multihot.pause_stats_collection_during_val_and_test(model)
+        print_memory("start transfer to multihot dataloader for dummy data ")
+        if train_dataloader:
+            train_dataloader = RestartableMap(
+                multihot.convert_to_multi_hot, train_dataloader
+            )
+
+        val_dataloader = RestartableMap(multihot.convert_to_multi_hot, val_dataloader)
+        test_dataloader = RestartableMap(multihot.convert_to_multi_hot, test_dataloader)
+
+    if args.ipex_optimize:
+        print_memory("start ipex_optimize ")
+        model.model, optimizer = ipex_optimize(args, model.model, optimizer, test_dataloader)
+
+    print_memory("start running model")
+    train_val_test(
+        args,
+        model,
+        optimizer,
+        device,
+        train_dataloader,
+        val_dataloader,
+        test_dataloader,
+        lr_scheduler,
+    )
+    if args.collect_multi_hot_freqs_stats:
+        multihot.save_freqs_stats()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/fp16/__init__.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/__init__.py
similarity index 80%
rename from models/image_recognition/tensorflow/vision_transformer/inference/fp16/__init__.py
rename to models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/__init__.py
index ce0a27f82..f7059a1a2 100644
--- a/models/image_recognition/tensorflow/vision_transformer/inference/fp16/__init__.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/__init__.py
@@ -1,20 +1,13 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2023 Intel Corporation
+# Copyright 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#    http://www.apache.org/licenses/LICENSE-2.0
+#     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-#
-
-#
-
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py
new file mode 100644
index 000000000..384f8fe89
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+from typing import List
+
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    #DAYS,
+    DEFAULT_CAT_NAMES,
+    DEFAULT_INT_NAMES,
+    InMemoryBinaryCriteoIterDataPipe,
+)
+# This is for crop dataset
+DAYS=1
+from torchrec.datasets.random import RandomRecDataset
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:multi_hot_criteo
+    from data.multi_hot_criteo import MultiHotCriteoIterDataPipe
+
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot_criteo import MultiHotCriteoIterDataPipe  # noqa F811
+except ImportError:
+    pass
+
+STAGES = ["train", "val", "test"]
+
+
+def _get_random_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    attr = f"limit_{stage}_batches"
+    num_batches = getattr(args, attr)
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    return DataLoader(
+        RandomRecDataset(
+            keys=DEFAULT_CAT_NAMES,
+            batch_size=batch_size,
+            hash_size=args.num_embeddings,
+            hash_sizes=args.num_embeddings_per_feature
+            if hasattr(args, "num_embeddings_per_feature")
+            else None,
+            manual_seed=args.seed if hasattr(args, "seed") else None,
+            ids_per_feature=1,
+            num_dense=len(DEFAULT_INT_NAMES),
+            num_batches=num_batches,
+        ),
+        batch_size=None,
+        batch_sampler=None,
+        pin_memory=args.pin_memory,
+        num_workers=0,
+    )
+
+
+def _get_in_memory_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    if args.in_memory_binary_criteo_path is not None:
+        dir_path = args.in_memory_binary_criteo_path
+        sparse_part = "sparse.npy"
+        datapipe = InMemoryBinaryCriteoIterDataPipe
+    else:
+        dir_path = args.synthetic_multi_hot_criteo_path
+        sparse_part = "sparse_multi_hot.npz"
+        datapipe = MultiHotCriteoIterDataPipe
+
+    if args.dataset_name == "criteo_kaggle":
+        # criteo_kaggle has no validation set, so use 2nd half of training set for now.
+        # Setting stage to "test" will get the 2nd half of the dataset.
+        # Setting root_name to "train" reads from the training set file.
+        (root_name, stage) = ("train", "test") if stage == "val" else stage
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"{root_name}_dense.npy")],
+            [os.path.join(dir_path, f"{root_name}_{sparse_part}")],
+            [os.path.join(dir_path, f"{root_name}_labels.npy")],
+        ]
+    # criteo_1tb code path uses below two conditionals
+    elif stage == "train":
+        stage_files: List[List[str]] = [
+            #[os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS - 1)],
+            #[os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS - 1)],
+            #[os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS - 1)],
+            # for crop dataset
+            [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS)],
+            [os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS)],
+            [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS)],
+        ]
+    elif stage in ["val", "test"]:
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"day_{DAYS-1}_dense.npy")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_{sparse_part}")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_labels.npy")],
+        ]
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    print(stage_files)
+    dataloader = DataLoader(
+        datapipe(
+            stage,
+            *stage_files,  # pyre-ignore[6]
+            batch_size=batch_size,
+            #rank=dist.get_rank(),
+            #world_size=dist.get_world_size(),
+            # The rand and world_size set for custom dist-dlrm
+            rank=0,
+            world_size=1,
+            drop_last=args.drop_last_training_batch if stage == "train" else False,
+            shuffle_batches=args.shuffle_batches,
+            shuffle_training_set=args.shuffle_training_set,
+            shuffle_training_set_random_seed=args.seed,
+            mmap_mode=args.mmap_mode,
+            hashes=args.num_embeddings_per_feature
+            if args.num_embeddings is None
+            else ([args.num_embeddings] * CAT_FEATURE_COUNT),
+        ),
+        batch_size=None,
+        pin_memory=args.pin_memory,
+        collate_fn=lambda x: x,
+    )
+    return dataloader
+
+
+def get_dataloader(args: argparse.Namespace, backend: str, stage: str) -> DataLoader:
+    """
+    Gets desired dataloader from dlrm_main command line options. Currently, this
+    function is able to return either a DataLoader wrapped around a RandomRecDataset or
+    a Dataloader wrapped around an InMemoryBinaryCriteoIterDataPipe.
+
+    Args:
+        args (argparse.Namespace): Command line options supplied to dlrm_main.py's main
+            function.
+        backend (str): "nccl" or "gloo".
+        stage (str): "train", "val", or "test".
+
+    Returns:
+        dataloader (DataLoader): PyTorch dataloader for the specified options.
+
+    """
+    stage = stage.lower()
+    if stage not in STAGES:
+        raise ValueError(f"Supplied stage was {stage}. Must be one of {STAGES}.")
+
+    args.pin_memory = (
+        (backend == "nccl") if not hasattr(args, "pin_memory") else args.pin_memory
+    )
+
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        return _get_random_dataloader(args, stage)
+    else:
+        return _get_in_memory_dataloader(args, stage)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py
new file mode 100644
index 000000000..cb520e2ec
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py
@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import zipfile
+from typing import Dict, Iterator, List, Optional
+
+import numpy as np
+import torch
+from iopath.common.file_io import PathManager, PathManagerFactory
+from pyre_extensions import none_throws
+from torch.utils.data import IterableDataset
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DEFAULT_CAT_NAMES,
+)
+from torchrec.datasets.utils import Batch, PATH_MANAGER_KEY
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class MultiHotCriteoIterDataPipe(IterableDataset):
+    """
+    Datapipe designed to operate over the MLPerf DLRM v2 synthetic multi-hot dataset.
+    This dataset can be created by following the steps in
+    torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py.
+    Each rank reads only the data for the portion of the dataset it is responsible for.
+
+    Args:
+        stage (str): "train", "val", or "test".
+        dense_paths (List[str]): List of path strings to dense npy files.
+        sparse_paths (List[str]): List of path strings to multi-hot sparse npz files.
+        labels_paths (List[str]): List of path strings to labels npy files.
+        batch_size (int): batch size.
+        rank (int): rank.
+        world_size (int): world size.
+        drop_last (Optional[bool]): Whether to drop the last batch if it is incomplete.
+        shuffle_batches (bool): Whether to shuffle batches
+        shuffle_training_set (bool): Whether to shuffle all samples in the dataset.
+        shuffle_training_set_random_seed (int): The random generator seed used when
+            shuffling the training set.
+        hashes (Optional[int]): List of max categorical feature value for each feature.
+            Length of this list should be CAT_FEATURE_COUNT.
+        path_manager_key (str): Path manager key used to load from different
+            filesystems.
+
+    Example::
+
+        datapipe = MultiHotCriteoIterDataPipe(
+            dense_paths=["day_0_dense.npy"],
+            sparse_paths=["day_0_sparse_multi_hot.npz"],
+            labels_paths=["day_0_labels.npy"],
+            batch_size=1024,
+            rank=torch.distributed.get_rank(),
+            world_size=torch.distributed.get_world_size(),
+        )
+        batch = next(iter(datapipe))
+    """
+
+    def __init__(
+        self,
+        stage: str,
+        dense_paths: List[str],
+        sparse_paths: List[str],
+        labels_paths: List[str],
+        batch_size: int,
+        rank: int,
+        world_size: int,
+        drop_last: Optional[bool] = False,
+        shuffle_batches: bool = False,
+        shuffle_training_set: bool = False,
+        shuffle_training_set_random_seed: int = 0,
+        mmap_mode: bool = False,
+        hashes: Optional[List[int]] = None,
+        path_manager_key: str = PATH_MANAGER_KEY,
+    ) -> None:
+        self.stage = stage
+        self.dense_paths = dense_paths
+        self.sparse_paths = sparse_paths
+        self.labels_paths = labels_paths
+        self.batch_size = batch_size
+        self.rank = rank
+        self.world_size = world_size
+        self.drop_last = drop_last
+        self.shuffle_batches = shuffle_batches
+        self.shuffle_training_set = shuffle_training_set
+        np.random.seed(shuffle_training_set_random_seed)
+        self.mmap_mode = mmap_mode
+        # hashes are not used because they were already applied in the
+        # script that generates the multi-hot dataset.
+        self.hashes: np.ndarray = np.array(hashes).reshape((CAT_FEATURE_COUNT, 1))
+        self.path_manager_key = path_manager_key
+        self.path_manager: PathManager = PathManagerFactory().get(path_manager_key)
+
+        if shuffle_training_set and stage == "train":
+            # Currently not implemented for the materialized multi-hot dataset.
+            self._shuffle_and_load_data_for_rank()
+        else:
+            m = "r" if mmap_mode else None
+            self.dense_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.dense_paths
+            ]
+            self.labels_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.labels_paths
+            ]
+            self.sparse_arrs: List = []
+            for sparse_path in self.sparse_paths:
+                multi_hot_ids_l = []
+                for feat_id_num in range(CAT_FEATURE_COUNT):
+                    multi_hot_ft_ids = self._load_from_npz(
+                        sparse_path, f"{feat_id_num}.npy"
+                    )
+                    multi_hot_ids_l.append(multi_hot_ft_ids)
+                self.sparse_arrs.append(multi_hot_ids_l)
+        len_d0 = len(self.dense_arrs[0])
+        second_half_start_index = int(len_d0 // 2 + len_d0 % 2)
+        if stage == "val":
+            self.dense_arrs[0] = self.dense_arrs[0][:second_half_start_index, :]
+            self.labels_arrs[0] = self.labels_arrs[0][:second_half_start_index, :]
+            self.sparse_arrs[0] = [
+                feats[:second_half_start_index, :] for feats in self.sparse_arrs[0]
+            ]
+        elif stage == "test":
+            self.dense_arrs[0] = self.dense_arrs[0][second_half_start_index:, :]
+            self.labels_arrs[0] = self.labels_arrs[0][second_half_start_index:, :]
+            self.sparse_arrs[0] = [
+                feats[second_half_start_index:, :] for feats in self.sparse_arrs[0]
+            ]
+        # When mmap_mode is enabled, sparse features are hashed when
+        # samples are batched in def __iter__. Otherwise, the dataset has been
+        # preloaded with sparse features hashed in the preload stage, here:
+        # if not self.mmap_mode and self.hashes is not None:
+        #     for k, _ in enumerate(self.sparse_arrs):
+        #         self.sparse_arrs[k] = [
+        #             feat % hash
+        #             for (feat, hash) in zip(self.sparse_arrs[k], self.hashes)
+        #         ]
+
+        self.num_rows_per_file: List[int] = list(map(len, self.dense_arrs))
+        total_rows = sum(self.num_rows_per_file)
+        self.num_full_batches: int = (
+            total_rows // batch_size // self.world_size * self.world_size
+        )
+        self.last_batch_sizes: np.ndarray = np.array(
+            [0 for _ in range(self.world_size)]
+        )
+        remainder = total_rows % (self.world_size * batch_size)
+        if not self.drop_last and 0 < remainder:
+            if remainder < self.world_size:
+                self.num_full_batches -= self.world_size
+                self.last_batch_sizes += batch_size
+            else:
+                self.last_batch_sizes += remainder // self.world_size
+            self.last_batch_sizes[: remainder % self.world_size] += 1
+
+        self.multi_hot_sizes: List[int] = [
+            multi_hot_feat.shape[-1] for multi_hot_feat in self.sparse_arrs[0]
+        ]
+
+        # These values are the same for the KeyedJaggedTensors in all batches, so they
+        # are computed once here. This avoids extra work from the KeyedJaggedTensor sync
+        # functions.
+        self.keys: List[str] = DEFAULT_CAT_NAMES
+        self.index_per_key: Dict[str, int] = {
+            key: i for (i, key) in enumerate(self.keys)
+        }
+
+    def _load_from_npz(self, fname, npy_name):
+        # figure out offset of .npy in .npz
+        zf = zipfile.ZipFile(fname)
+        info = zf.NameToInfo[npy_name]
+        assert info.compress_type == 0
+        zf.fp.seek(info.header_offset + len(info.FileHeader()) + 20)
+        # read .npy header
+        zf.open(npy_name, "r")
+        version = np.lib.format.read_magic(zf.fp)
+        shape, fortran_order, dtype = np.lib.format._read_array_header(zf.fp, version)
+        assert (
+            dtype == "int32"
+        ), f"sparse multi-hot dtype is {dtype} but should be int32"
+        offset = zf.fp.tell()
+        # create memmap
+        return np.memmap(
+            zf.filename,
+            dtype=dtype,
+            shape=shape,
+            order="F" if fortran_order else "C",
+            mode="r",
+            offset=offset,
+        )
+
+    def _np_arrays_to_batch(
+        self,
+        dense: np.ndarray,
+        sparse: List[np.ndarray],
+        labels: np.ndarray,
+    ) -> Batch:
+        if self.shuffle_batches:
+            # Shuffle all 3 in unison
+            shuffler = np.random.permutation(len(dense))
+            sparse = [multi_hot_ft[shuffler, :] for multi_hot_ft in sparse]
+            dense = dense[shuffler]
+            labels = labels[shuffler]
+
+        batch_size = len(dense)
+        lengths = torch.ones((CAT_FEATURE_COUNT * batch_size), dtype=torch.int32)
+        for k, multi_hot_size in enumerate(self.multi_hot_sizes):
+            lengths[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        offsets = torch.cumsum(torch.concat((torch.tensor([0]), lengths)), dim=0)
+        length_per_key = [
+            batch_size * multi_hot_size for multi_hot_size in self.multi_hot_sizes
+        ]
+        offset_per_key = torch.cumsum(
+            torch.concat((torch.tensor([0]), torch.tensor(length_per_key))), dim=0
+        )
+        values = torch.concat([torch.from_numpy(feat).flatten() for feat in sparse])
+        return Batch(
+            dense_features=torch.from_numpy(dense.copy()),
+            sparse_features=KeyedJaggedTensor(
+                keys=self.keys,
+                values=values,
+                lengths=lengths,
+                offsets=offsets,
+                stride=batch_size,
+                length_per_key=length_per_key,
+                offset_per_key=offset_per_key.tolist(),
+                index_per_key=self.index_per_key,
+            ),
+            labels=torch.from_numpy(labels.reshape(-1).copy()),
+        )
+
+    def __iter__(self) -> Iterator[Batch]:
+        # Invariant: buffer never contains more than batch_size rows.
+        buffer: Optional[List[np.ndarray]] = None
+
+        def append_to_buffer(
+            dense: np.ndarray,
+            sparse: List[np.ndarray],
+            labels: np.ndarray,
+        ) -> None:
+            nonlocal buffer
+            if buffer is None:
+                buffer = [dense, sparse, labels]
+            else:
+                buffer[0] = np.concatenate((buffer[0], dense))
+                buffer[1] = [np.concatenate((b, s)) for b, s in zip(buffer[1], sparse)]
+                buffer[2] = np.concatenate((buffer[2], labels))
+
+        # Maintain a buffer that can contain up to batch_size rows. Fill buffer as
+        # much as possible on each iteration. Only return a new batch when batch_size
+        # rows are filled.
+        file_idx = 0
+        row_idx = 0
+        batch_idx = 0
+        buffer_row_count = 0
+        cur_batch_size = (
+            self.batch_size if self.num_full_batches > 0 else self.last_batch_sizes[0]
+        )
+        while (
+            batch_idx
+            < self.num_full_batches + (self.last_batch_sizes[0] > 0) * self.world_size
+        ):
+            if buffer_row_count == cur_batch_size or file_idx == len(self.dense_arrs):
+                if batch_idx % self.world_size == self.rank:
+                    yield self._np_arrays_to_batch(*none_throws(buffer))
+                    buffer = None
+                buffer_row_count = 0
+                batch_idx += 1
+                if (
+                    0 <= batch_idx - self.num_full_batches < self.world_size
+                    and (self.last_batch_sizes[0] > 0)
+                ):
+                    cur_batch_size = self.last_batch_sizes[
+                        batch_idx - self.num_full_batches
+                    ]
+            else:
+                rows_to_get = min(
+                    cur_batch_size - buffer_row_count,
+                    self.num_rows_per_file[file_idx] - row_idx,
+                )
+                buffer_row_count += rows_to_get
+                slice_ = slice(row_idx, row_idx + rows_to_get)
+
+                if batch_idx % self.world_size == self.rank:
+                    dense_inputs = self.dense_arrs[file_idx][slice_, :]
+                    sparse_inputs = [
+                        feats[slice_, :] for feats in self.sparse_arrs[file_idx]
+                    ]
+                    target_labels = self.labels_arrs[file_idx][slice_, :]
+
+                    # if self.mmap_mode and self.hashes is not None:
+                    #     sparse_inputs = [
+                    #         feats % hash
+                    #         for (feats, hash) in zip(sparse_inputs, self.hashes)
+                    #     ]
+
+                    append_to_buffer(
+                        dense_inputs,
+                        sparse_inputs,
+                        target_labels,
+                    )
+                row_idx += rows_to_get
+
+                if row_idx >= self.num_rows_per_file[file_idx]:
+                    file_idx += 1
+                    row_idx = 0
+
+    def __len__(self) -> int:
+        return self.num_full_batches // self.world_size + (self.last_batch_sizes[0] > 0)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dist_models.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dist_models.py
new file mode 100644
index 000000000..eff746a8a
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dist_models.py
@@ -0,0 +1,241 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import torch
+from torchrec import EmbeddingBagCollection
+import extend_distributed as ext_dist
+from torchrec.models.dlrm import DLRM, InteractionDCNArch, OverArch
+from typing import Dict, List, Optional, Tuple
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from torchrec.modules.crossnet import LowRankCrossNet
+import time
+
+class DIST_DLRM_DCN(DLRM):
+    """
+    Recsys model with DCN modified from the original model from "Deep Learning Recommendation
+    Model for Personalization and Recommendation Systems"
+    (https://arxiv.org/abs/1906.00091). Similar to DLRM module but has
+    DeepCrossNet https://arxiv.org/pdf/2008.13535.pdf as the interaction layer.
+
+    The module assumes all sparse features have the same embedding dimension
+    (i.e. each EmbeddingBagConfig uses the same embedding_dim).
+
+    The following notation is used throughout the documentation for the models:
+
+    * F: number of sparse features
+    * D: embedding_dimension of sparse features
+    * B: batch size
+    * num_features: number of dense features
+
+    Args:
+        embedding_bag_collection (EmbeddingBagCollection): collection of embedding bags
+            used to define `SparseArch`.
+        dense_in_features (int): the dimensionality of the dense input features.
+        dense_arch_layer_sizes (List[int]): the layer sizes for the `DenseArch`.
+        over_arch_layer_sizes (List[int]): the layer sizes for the `OverArch`.
+            The output dimension of the `InteractionArch` should not be manually
+            specified here.
+        dcn_num_layers (int): the number of DCN layers in the interaction.
+        dcn_low_rank_dim (int): the dimensionality of low rank approximation
+            used in the dcn layers.
+        dense_device (Optional[torch.device]): default compute device.
+
+    Example::
+
+        B = 2
+        D = 8
+
+        eb1_config = EmbeddingBagConfig(
+           name="t1", embedding_dim=D, num_embeddings=100, feature_names=["f1", "f3"]
+        )
+        eb2_config = EmbeddingBagConfig(
+           name="t2",
+           embedding_dim=D,
+           num_embeddings=100,
+           feature_names=["f2"],
+        )
+
+        ebc = EmbeddingBagCollection(tables=[eb1_config, eb2_config])
+        model = DLRM_DCN(
+           embedding_bag_collection=ebc,
+           dense_in_features=100,
+           dense_arch_layer_sizes=[20, D],
+           dcn_num_layers=2,
+           dcn_low_rank_dim=8,
+           over_arch_layer_sizes=[5, 1],
+        )
+
+        features = torch.rand((B, 100))
+
+        #     0       1
+        # 0   [1,2] [4,5]
+        # 1   [4,3] [2,9]
+        # ^
+        # feature
+        sparse_features = KeyedJaggedTensor.from_offsets_sync(
+           keys=["f1", "f3"],
+           values=torch.tensor([1, 2, 4, 5, 4, 3, 2, 9]),
+           offsets=torch.tensor([0, 2, 4, 6, 8]),
+        )
+
+        logits = model(
+           dense_features=features,
+           sparse_features=sparse_features,
+        )
+    """
+
+    def __init__(
+        self,
+        embedding_bag_collection: EmbeddingBagCollection,
+        dense_in_features: int,
+        num_total_sparse_features:int,
+        dense_arch_layer_sizes: List[int],
+        over_arch_layer_sizes: List[int],
+        dcn_num_layers: int,
+        dcn_low_rank_dim: int,
+        n_emb : int,
+        shard_matrix,
+        shard_strategy,
+        args: argparse.Namespace,
+        dense_device: Optional[torch.device] = None,
+    ) -> None:
+        # initialize DLRM
+        # sparse arch and dense arch are initialized via DLRM
+        super().__init__(
+            embedding_bag_collection,
+            dense_in_features,
+            dense_arch_layer_sizes,
+            over_arch_layer_sizes,
+            dense_device,
+        )
+        
+        self.args = args
+        if ext_dist.my_size > 1:
+            if n_emb < ext_dist.my_size:
+                sys.exit(
+                    "only (%d) sparse features for (%d) devices, table partitions will fail"
+                    % (n_emb, ext_dist.my_size)
+                )
+            self.n_global_emb = n_emb
+            #self.n_local_emb, self.n_emb_per_rank = ext_dist.get_split_lengths(
+            #    n_emb
+            #)
+            self.shard_matrix = shard_matrix
+            self.shard_strategy = shard_strategy
+            self.n_emb_per_rank = [len(s_m) for s_m in shard_matrix]
+            #self.local_emb_slice = ext_dist.get_my_slice(n_emb)
+            #self.local_emb_indices = list(range(n_emb))[self.local_emb_slice]
+
+
+        self.embedding_dim: int = embedding_bag_collection.embedding_bag_configs()[
+            0
+        ].embedding_dim
+        #num_sparse_features: int = len(self.sparse_arch.sparse_feature_names)
+
+        # Fix interaction and over arch for DLRM_DCN
+
+        crossnet = LowRankCrossNet(
+            in_features=(num_total_sparse_features + 1) * self.embedding_dim,
+            num_layers=dcn_num_layers,
+            low_rank=dcn_low_rank_dim,
+        )
+
+        self.inter_arch = InteractionDCNArch(
+            num_sparse_features=num_total_sparse_features,
+            crossnet=crossnet,
+        )
+        self.inter_arch.to(dense_device)
+
+        over_in_features: int = (num_total_sparse_features + 1) * self.embedding_dim
+
+        self.over_arch = OverArch(
+            in_features=over_in_features,
+            layer_sizes=over_arch_layer_sizes,
+            device=dense_device,
+        )
+
+    def sequential_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = self.sparse_arch(sparse_features)
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def distributed_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_sparse = self.sparse_arch(sparse_features)
+        a2a_req = ext_dist.alltoall(embedded_sparse, self.n_emb_per_rank)
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = a2a_req.wait()
+        #['0', '8', '16', '24'], ['1', '9', '17', '25'], ['2', '10', '18'], ['3', '11', '19'], ['4', '12', '20'], ['5', '13', '21'], ['6', '14', '22'], ['7', '15', '23']
+        split_embed = {}
+        for index_rank in range(len(self.shard_matrix)):
+            embed_rank = torch.split(embedded_sparse[index_rank], self.embedding_dim, dim=1)
+            for index in range(len(self.shard_matrix[index_rank])):
+                split_embed[self.shard_matrix[index_rank][index]] = embed_rank[index]
+
+        embedded_sparse = []
+        for i in range(len(self.args.num_embeddings_per_feature)):
+            embedded_sparse.append(split_embed[i])
+        embedded_sparse = torch.cat(list(embedded_sparse), dim=1).reshape(embedded_dense.size(0), -1, embedded_dense.size(-1))
+        if self.args.fp16:
+            embedded_sparse = embedded_sparse.half() if embedded_sparse.dtype != torch.float16 else embedded_sparse
+            embedded_dense = embedded_dense.half() if embedded_dense.dtype != torch.float16 else embedded_dense
+        elif self.args.bf16:
+            embedded_sparse = embedded_sparse.bfloat16() if embedded_sparse.dtype != torch.bfloat16 else embedded_sparse
+            embedded_dense = embedded_dense.bfloat16() if embedded_dense.dtype != torch.bfloat16 else embedded_dense
+
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        if ext_dist.my_size > 1:
+            # multi-node multi-device run
+            return self.distributed_forward(dense_features, sparse_features)
+        else:
+            # single device run
+            return self.sequential_forward(dense_features, sparse_features)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py
new file mode 100644
index 000000000..c65c14bfa
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py
@@ -0,0 +1,1055 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import argparse
+import itertools
+import os
+import sys
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import List, Optional
+import numpy as np
+
+import torch
+import torchmetrics as metrics
+from torch.autograd.profiler import record_function
+from pyre_extensions import none_throws
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec import EmbeddingBagCollection
+from torchrec.datasets.criteo import DEFAULT_CAT_NAMES, DEFAULT_INT_NAMES
+#from torchrec.distributed import TrainPipelineSparseDist
+from torchrec.distributed.comm import get_local_size
+#from torchrec.distributed.model_parallel import (
+#    DistributedModelParallel,
+#    get_default_sharders,
+#)
+#from torchrec.distributed.planner import EmbeddingShardingPlanner, Topology
+#from torchrec.distributed.planner.storage_reservations import (
+#    HeuristicalStorageReservation,
+#)
+from dist_models import DIST_DLRM_DCN
+from torchrec.models.dlrm import DLRM, DLRM_DCN, DLRM_Projection, DLRMTrain
+from torchrec.modules.embedding_configs import EmbeddingBagConfig
+from torchrec.optim.apply_optimizer_in_backward import apply_optimizer_in_backward
+from torchrec.optim.keyed import CombinedOptimizer, KeyedOptimizerWrapper
+from torchrec.optim.optimizers import in_backward_optimizer_filter
+from tqdm import tqdm
+import extend_distributed as ext_dist
+from extend_distributed import dist, env2int
+from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
+
+import logging
+import sharding
+
+logger: logging.Logger = logging.getLogger(__name__)
+logger.setLevel(1)
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:dlrm_dataloader
+    from data.dlrm_dataloader import get_dataloader
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:lr_scheduler
+    from lr_scheduler import LRPolicyScheduler
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot, RestartableMap
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .data.dlrm_dataloader import get_dataloader  # noqa F811
+    from .lr_scheduler import LRPolicyScheduler  # noqa F811
+    from .multi_hot import Multihot, RestartableMap  # noqa F811
+except ImportError:
+    pass
+
+TRAIN_PIPELINE_STAGES = 3  # Number of stages in TrainPipelineSparseDist.
+
+n_emb = None
+#local_emb_slice = None
+local_emb_indices = None
+default_embedding_names = None
+idx_name = None
+
+
+class InteractionType(Enum):
+    ORIGINAL = "original"
+    DCN = "dcn"
+    PROJECTION = "projection"
+
+    def __str__(self):
+        return self.value
+
+
+def load_snapshot(model, snapshot_dir):
+    from torchsnapshot import Snapshot
+    snapshot = Snapshot(path=snapshot_dir)
+    snapshot.restore(app_state={"model": model})
+
+
+def parse_args(argv: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
+    parser.add_argument(
+        "--epochs",
+        type=int,
+        default=1,
+        help="number of epochs to train",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=32,
+        help="batch size to use for training",
+    )
+    parser.add_argument(
+        "--drop_last_training_batch",
+        dest="drop_last_training_batch",
+        action="store_true",
+        help="Drop the last non-full training batch",
+    )
+    parser.add_argument(
+        "--test_batch_size",
+        type=int,
+        default=None,
+        help="batch size to use for validation and testing",
+    )
+    parser.add_argument(
+        "--limit_train_batches",
+        type=int,
+        default=None,
+        help="number of train batches",
+    )
+    parser.add_argument(
+        "--limit_val_batches",
+        type=int,
+        default=None,
+        help="number of validation batches",
+    )
+    parser.add_argument(
+        "--limit_test_batches",
+        type=int,
+        default=None,
+        help="number of test batches",
+    )
+    parser.add_argument(
+        "--dataset_name",
+        type=str,
+        choices=["criteo_1t", "criteo_kaggle"],
+        default="criteo_1t",
+        help="dataset for experiment, current support criteo_1tb, criteo_kaggle",
+    )
+    parser.add_argument(
+        "--num_embeddings",
+        type=int,
+        default=100_000,
+        help="max_ind_size. The number of embeddings in each embedding table. Defaults"
+        " to 100_000 if num_embeddings_per_feature is not supplied.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        default=None,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--dense_arch_layer_sizes",
+        type=str,
+        default="512,256,64",
+        help="Comma separated layer sizes for dense arch.",
+    )
+    parser.add_argument(
+        "--over_arch_layer_sizes",
+        type=str,
+        default="512,512,256,1",
+        help="Comma separated layer sizes for over arch.",
+    )
+    parser.add_argument(
+        "--embedding_dim",
+        type=int,
+        default=64,
+        help="Size of each embedding.",
+    )
+    parser.add_argument(
+        "--interaction_branch1_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch1 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--interaction_branch2_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch2 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--dcn_num_layers",
+        type=int,
+        default=3,
+        help="Number of DCN layers in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--dcn_low_rank_dim",
+        type=int,
+        default=512,
+        help="Low rank dimension for DCN in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--undersampling_rate",
+        type=float,
+        help="Desired proportion of zero-labeled samples to retain (i.e. undersampling zero-labeled rows)."
+        " Ex. 0.3 indicates only 30pct of the rows with label 0 will be kept."
+        " All rows with label 1 will be kept. Value should be between 0 and 1."
+        " When not supplied, no undersampling occurs.",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        help="Random seed for reproducibility.",
+    )
+    parser.add_argument(
+        "--pin_memory",
+        dest="pin_memory",
+        action="store_true",
+        help="Use pinned memory when loading data.",
+    )
+    parser.add_argument(
+        "--mmap_mode",
+        dest="mmap_mode",
+        action="store_true",
+        help="--mmap_mode mmaps the dataset."
+        " That is, the dataset is kept on disk but is accessed as if it were in memory."
+        " --mmap_mode is intended mostly for faster debugging. Use --mmap_mode to bypass"
+        " preloading the dataset when preloading takes too long or when there is "
+        " insufficient memory available to load the full dataset.",
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the Criteo dataset npy files.",
+    )
+    parser.add_argument(
+        "--synthetic_multi_hot_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the MLPerf v2 synthetic multi-hot dataset npz files.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=15.0,
+        help="Learning rate.",
+    )
+    parser.add_argument(
+        "--eps",
+        type=float,
+        default=1e-8,
+        help="Epsilon for Adagrad optimizer.",
+    )
+    parser.add_argument(
+        "--shuffle_batches",
+        dest="shuffle_batches",
+        action="store_true",
+        help="Shuffle each batch during training.",
+    )
+    parser.add_argument(
+        "--shuffle_training_set",
+        dest="shuffle_training_set",
+        action="store_true",
+        help="Shuffle the training set in memory. This will override mmap_mode",
+    )
+    parser.add_argument(
+        "--validation_freq_within_epoch",
+        type=int,
+        default=None,
+        help="Frequency at which validation will be run within an epoch.",
+    )
+    parser.set_defaults(
+        pin_memory=None,
+        mmap_mode=None,
+        drop_last=None,
+        shuffle_batches=None,
+        shuffle_training_set=None,
+    )
+    parser.add_argument(
+        "--adagrad",
+        dest="adagrad",
+        action="store_true",
+        help="Flag to determine if adagrad optimizer should be used.",
+    )
+    parser.add_argument(
+        "--interaction_type",
+        type=InteractionType,
+        choices=list(InteractionType),
+        default=InteractionType.ORIGINAL,
+        help="Determine the interaction type to be used (original, dcn, or projection)"
+        " default is original DLRM with pairwise dot product",
+    )
+    parser.add_argument(
+        "--collect_multi_hot_freqs_stats",
+        dest="collect_multi_hot_freqs_stats",
+        action="store_true",
+        help="Flag to determine whether to collect stats on freq of embedding access.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        default=None,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default=None,
+        help="Multi-hot distribution options.",
+    )
+    parser.add_argument("--lr_warmup_steps", type=int, default=0)
+    parser.add_argument("--lr_decay_start", type=int, default=0)
+    parser.add_argument("--lr_decay_steps", type=int, default=0)
+    parser.add_argument(
+        "--print_lr",
+        action="store_true",
+        help="Print learning rate every iteration.",
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help="Enable TensorFloat-32 mode for matrix multiplications on A100 (or newer) GPUs.",
+    )
+    parser.add_argument(
+        "--print_sharding_plan",
+        action="store_true",
+        help="Print the sharding plan used for each embedding table.",
+    )
+    parser.add_argument("--use_xpu", action="store_true", default=False)
+    parser.add_argument(
+        "--log_freq",
+        type=int,
+        default=0,
+        help="log_freq to print performance statistics.",
+    )
+    parser.add_argument("--numpy_rand_seed", type=int, default=123)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--fp16", action="store_true", default=False)
+    parser.add_argument("--amp", action="store_true", default=False)
+    parser.add_argument("--inference_only", action="store_true", default=False)
+    parser.add_argument("--train_with_val", action="store_true", default=False)
+    parser.add_argument("--num_batches", type=int, default=0)
+    parser.add_argument("--num_nodes", type=int, default=1)
+    parser.add_argument("--trans_data", action="store_true", default=False)
+    parser.add_argument(
+    "--sharding_plan",
+    help="Sharding plan to use",
+    type=str,
+    choices=["round_robin", "uniform", "auto", "hier_auto", "custom", "max_min"],
+    default="round_robin",)
+    parser.add_argument(
+        "--snapshot_dir",
+        help="Weight path for dlrm-v2",
+        type=str,
+    )
+
+
+    return parser.parse_args(argv)
+
+
+def _evaluate(
+    limit_batches: Optional[int],
+    #eval_pipeline: TrainPipelineSparseDist,
+    eval_model,
+    eval_dataloader: DataLoader,
+    stage: str,
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> float:
+    """
+    Evaluates model. Computes and prints AUROC. Helper function for train_val_test.
+
+    Args:
+        limit_batches (Optional[int]): Limits the dataloader to the first `limit_batches` batches.
+        eval_pipeline (TrainPipelineSparseDist): pipelined model.
+        eval_dataloader (DataLoader): Dataloader for either the validation set or test set.
+        stage (str): "val" or "test".
+
+    Returns:
+        float: auroc result
+    """
+    eval_model.eval()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+
+    def eval_step(model, batch):
+        batch_size = batch.dense_features.size(0)
+        if args.amp:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        batch.labels = batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+        for k, v in batch.sparse_features.items():
+            batch.sparse_features[k]['values'] = batch.sparse_features[k]['values'].to(device)
+            batch.sparse_features[k]['offsets'] = batch.sparse_features[k]['offsets'].to(device)
+
+        if args.use_device == "xpu":
+            with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                logits = model(batch.dense_features, batch.sparse_features)
+        elif args.use_device == "cuda":
+            with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                logits = model(batch.dense_features, batch.sparse_features)
+        else:
+            pass
+        return logits, batch.labels
+    # Set eval_pipeline._connected to False to cause the pipeline to refill with new batches as if it were newly created and empty.
+    #eval_pipeline._connected = False
+
+    iterator = itertools.islice(iter(eval_dataloader), limit_batches)
+
+    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+
+    is_rank_zero = dist.get_rank() == 0
+    #if is_rank_zero:
+    #    pbar = tqdm(
+    #        iter(int, 1),
+    #        desc=f"Evaluating {stage} set",
+    #        total=len(eval_dataloader),
+    #        disable=False,
+    #    )
+    total_t = 0
+    it = 1
+    with torch.no_grad():
+        while True:
+            try:
+                batch = next(iterator)
+                batch.sparse_features = unpack(batch.sparse_features, default_embedding_names)
+                ext_dist.barrier()
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+                t1 = time.time()
+                with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                    logits, labels = eval_step(eval_model, batch)
+                    #_loss, logits, labels = eval_pipeline.progress(iterator)
+                    preds = torch.sigmoid(logits)
+                    auroc(preds, labels)
+                    if args.use_device == "xpu":
+                        torch.xpu.synchronize()
+                    elif args.use_device == "cuda":
+                        torch.cuda.synchronize()
+                    else:
+                        pass
+                    if it > 5 and args.inference_only:
+                        total_t += time.time() - t1
+                        #if log_freq != 0 and it % log_freq == 0:
+                        logger.info(f"avg eval time per iter at ITER: {it - 5}, {total_t/(it-5)} s")
+            except StopIteration:
+                break
+            if args.enable_profiling and it == 35 and args.inference_only:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_inference_' + rank_index + 'profiling'
+                if args.use_xpu:
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            #if is_rank_zero:
+            #    pbar.update(1)
+            it += 1
+            if it > args.num_batches:
+                break;
+
+    auroc_result = auroc.compute().item()
+    num_samples = torch.tensor(sum(map(len, auroc.target)), device=device)
+    dist.reduce(num_samples, 0, op=dist.ReduceOp.SUM)
+
+    if is_rank_zero:
+        print(f"AUROC over {stage} set: {auroc_result}.")
+        print(f"Number of {stage} samples: {num_samples}")
+    return auroc_result
+
+def _train(
+    train_model,
+    train_optimizer,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    epoch: int,
+    lr_scheduler,
+    print_lr: bool,
+    validation_freq: Optional[int],
+    limit_train_batches: Optional[int],
+    limit_val_batches: Optional[int],
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> bool:
+    """
+    Trains model for 1 epoch. Helper function for train_val_test.
+    Args:
+        train_pipeline (TrainPipelineSparseDist): pipelined model used for training.
+        val_pipeline (TrainPipelineSparseDist): pipelined model used for validation.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        epoch (int): The number of complete passes through the training set so far.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+        print_lr (bool): Whether to print the learning rate every training step.
+        validation_freq (Optional[int]): The number of training steps between validation runs within an epoch.
+        limit_train_batches (Optional[int]): Limits the training set to the first `limit_train_batches` batches.
+        limit_val_batches (Optional[int]): Limits the validation set to the first `limit_val_batches` batches.
+        print_progress (bool): Whether to print tqdm progress bar.
+    Returns:
+        bool: Whether the validation_auroc threshold is reached.
+    """
+    train_model.train()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    def train_step(model, opt, next_batch):
+        batch_size = next_batch.dense_features.size(0)
+        opt.zero_grad(set_to_none=True)
+        if args.amp:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        next_batch.labels = next_batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+
+        for k, v in next_batch.sparse_features.items():
+            next_batch.sparse_features[k]['values'] = next_batch.sparse_features[k]['values'].to(device)
+            next_batch.sparse_features[k]['offsets'] = next_batch.sparse_features[k]['offsets'].to(device)
+        if args.use_device == "xpu":
+            with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                losses, _ = model(next_batch)
+                loss = torch.sum(losses, dim=0)
+        elif args.use_device == "cuda":
+            with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                losses, _ = model(next_batch)
+                loss = torch.sum(losses, dim=0)
+        else:
+            pass
+        loss.backward()
+        opt.step()
+    iterator = itertools.islice(iter(train_dataloader), limit_train_batches)
+
+    is_rank_zero = dist.get_rank() == 0
+    #if is_rank_zero:
+    #    pbar = tqdm(
+    #        iter(int, 1),
+    #        desc=f"Epoch {epoch}",
+    #        total=len(train_dataloader),
+    #        disable=False,
+    #    )
+
+    total_t = 0
+    for it in itertools.count(1):
+        try:
+            if is_rank_zero and print_lr:
+                for i, g in enumerate(train_pipeline._optimizer.param_groups):
+                    print(f"lr: {it} {i} {g['lr']:.6f}")
+            next_batch = next(iterator)
+            next_batch.sparse_features = unpack(next_batch.sparse_features, default_embedding_names)
+            ext_dist.barrier()
+            if args.use_device == "xpu":
+                torch.xpu.synchronize()
+            elif args.use_device == "cuda":
+                torch.cuda.synchronize()
+            else:
+                pass
+            t1 = time.time()
+            with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True, record_shapes=False) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                train_step(train_model, train_optimizer, next_batch)
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+
+                if it > 5:
+                    total_t += time.time() - t1
+                    #if log_freq != 0 and it % log_freq == 0:
+                    logger.info(f"avg training time per iter at ITER: {it - 5}, {total_t/(it - 5)} s")
+
+            if args.enable_profiling and it == 35:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_training_' + rank_index + 'profiling'
+                if args.use_device == "xpu":
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            lr_scheduler.step()
+            #if is_rank_zero:
+            #    pbar.update(1)
+            if validation_freq and (it % validation_freq == 0) and args.train_with_val:
+                auroc_result = _evaluate(
+                    limit_val_batches,
+                    train_model.model,
+                    val_dataloader,
+                    "val",
+                    log_freq, 
+                    device, 
+                    args
+                )
+                if is_rank_zero:
+                    logger.info("The val auc = {auroc_result}")
+                train_model.train()
+        except StopIteration:
+            # Dataset traversal complete
+            break
+        it += 1
+        if it > args.num_batches:
+            break;
+
+    logger.info(f"Total number of iterations: {it - 1}")
+
+
+@dataclass
+class TrainValTestResults:
+    val_aurocs: List[float] = field(default_factory=list)
+    test_auroc: Optional[float] = None
+
+
+def train_val_test(
+    args: argparse.Namespace,
+    model: torch.nn.Module,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    test_dataloader: DataLoader,
+    lr_scheduler: LRPolicyScheduler,
+) -> TrainValTestResults:
+    """
+    Train/validation/test loop.
+
+    Args:
+        args (argparse.Namespace): parsed command line args.
+        model (torch.nn.Module): model to train.
+        optimizer (torch.optim.Optimizer): optimizer to use.
+        device (torch.device): device to use.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        test_dataloader (DataLoader): Test set's dataloader.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+
+    Returns:
+        TrainValTestResults.
+    """
+    results = TrainValTestResults()
+    #train_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    #val_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    #test_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    if args.fp16 and (not args.amp):
+        model.to(torch.float16)
+    if args.bf16 and (not args.amp):
+        model.to(torch.bfloat16)
+
+    if not args.inference_only:
+        for epoch in range(args.epochs):
+            _train(
+                model,
+                optimizer,
+                train_dataloader,
+                val_dataloader,
+                epoch,
+                lr_scheduler,
+                args.print_lr,
+                args.validation_freq_within_epoch,
+                args.limit_train_batches,
+                args.limit_val_batches,
+                args.log_freq,
+                device,
+                args
+            )
+            if args.train_with_val:
+                val_auroc = _evaluate(
+                    args.limit_val_batches, model.model, val_dataloader, "val", args.log_freq, device, args
+                )
+                results.val_aurocs.append(val_auroc)
+
+    else:
+        test_auroc = _evaluate(
+            args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+        )
+        results.test_auroc = test_auroc
+
+    return results
+
+
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+    args = parse_args(argv)
+    args.amp_dtype = torch.float32
+    if args.bf16:
+        args.amp_dtype = torch.bfloat16
+    if args.fp16:
+        args.amp_dtype = torch.float16
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    args.enable_profiling = False
+    if os.getenv('PROFILE') == '1': args.enable_profiling=True
+
+    if args.use_xpu:
+        import intel_extension_for_pytorch
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError as e:
+            pass
+
+    if not args.use_xpu:
+        torch.backends.cuda.matmul.allow_tf32 = args.allow_tf32
+
+    if args.multi_hot_sizes is not None and args.trans_data:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or (not args.trans_data) or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    np.random.seed(args.numpy_rand_seed)
+    torch.manual_seed(args.numpy_rand_seed)
+
+    #rank = int(os.environ["LOCAL_RANK"])
+    rank = env2int(
+        [
+            "MPI_LOCALRANKID",
+            "OMPI_COMM_WORLD_LOCAL_RANK",
+            "MV2_COMM_WORLD_LOCAL_RANK",
+            "LOCAL_RANK",
+        ],
+        0,
+    )
+    args.use_device = "cpu"
+    if args.use_xpu and torch.xpu.is_available():
+        torch.xpu.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device("xpu", rank)
+        backend = "ccl"
+        torch.xpu.set_device(device)
+        args.use_device = "xpu"
+    elif torch.cuda.is_available():
+        torch.cuda.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device(f"cuda:{rank}")
+        backend = "nccl"
+        torch.cuda.set_device(device)
+        args.use_device = "cuda"
+    else:
+        device: torch.device = torch.device("cpu")
+        backend = "gloo"
+
+    if rank == 0:
+        print(
+            "PARAMS: (lr, batch_size, warmup_steps, decay_start, decay_steps): "
+            f"{(args.learning_rate, args.batch_size, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps)}"
+        )
+    ext_dist.init_distributed(
+        local_rank=-1, use_device=args.use_device, backend=backend
+    )
+
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path
+        is args.synthetic_multi_hot_criteo_path
+        is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    train_dataloader = get_dataloader(args, backend, "train")
+    val_dataloader = get_dataloader(args, backend, "val")
+    test_dataloader = get_dataloader(args, backend, "test")
+    # use dist_models, if not you can comment out
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    n_emb = len(DEFAULT_CAT_NAMES)
+    #if dist.get_world_size() == 8 and dist.get_rank() == 6:
+    #    local_emb_indices[1] = 25
+    #if dist.get_world_size() == 8 and dist.get_rank() == 7:
+    #    local_emb_indices[2] = 21
+
+    shard_matrix_str, shard_strategy_str, shard_matrix, shard_strategy = sharding.generate_plan(
+    args.num_embeddings_per_feature, args.multi_hot_sizes, args.num_nodes, dist.get_world_size(), args, dist.get_rank() == 0)
+    #local_emb_slice = ext_dist.get_my_slice(n_emb)
+    local_emb_indices = shard_matrix[rank]
+    default_embedding_names = [DEFAULT_CAT_NAMES[i] for i in local_emb_indices]
+    idx_name = zip(local_emb_indices, default_embedding_names)
+
+    eb_configs = [
+        EmbeddingBagConfig(
+            name=f"t_{feature_name}",
+            embedding_dim=args.embedding_dim,
+            num_embeddings=none_throws(args.num_embeddings_per_feature)[feature_idx]
+            if args.num_embeddings is None
+            else args.num_embeddings,
+            feature_names=[feature_name],
+        )
+        #for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
+        for feature_idx, feature_name in idx_name
+    ]
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    if args.interaction_type == InteractionType.ORIGINAL:
+        dlrm_model = DLRM(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.DCN:
+        #dlrm_model = DLRM_DCN(
+        dlrm_model = DIST_DLRM_DCN(
+            embedding_bag_collection=EmbeddingBagCollection(
+                #tables=eb_configs, device=torch.device("meta")
+                tables=eb_configs, device="cpu"
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            num_total_sparse_features=n_emb,
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dcn_num_layers=args.dcn_num_layers,
+            dcn_low_rank_dim=args.dcn_low_rank_dim,
+            n_emb=n_emb,
+            shard_matrix=shard_matrix, 
+            shard_strategy = shard_strategy,
+            args=args,
+            dense_device="cpu",
+        )
+    elif args.interaction_type == InteractionType.PROJECTION:
+        dlrm_model = DLRM_Projection(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
+            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
+            dense_device=device,
+        )
+    else:
+        raise ValueError(
+            "Unknown interaction option set. Should be original, dcn, or projection."
+        )
+
+    if not args.amp:
+        dlrm_model.to(args.amp_dtype)
+    train_model = DLRMTrain(dlrm_model)
+    if args.inference_only:
+        assert args.snapshot_dir
+        load_snapshot(train_model, args.snapshot_dir)
+
+    #embedding_optimizer = torch.optim.Adagrad if args.adagrad else torch.optim.SGD
+    # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
+    # the optimizer update will be applied in the backward pass, in this case through a fused op.
+    # TorchRec will use the FBGEMM implementation of EXACT_ADAGRAD. For GPU devices, a fused CUDA kernel is invoked. For CPU, FBGEMM_GPU invokes CPU kernels
+    # https://github.com/pytorch/FBGEMM/blob/2cb8b0dff3e67f9a009c4299defbd6b99cc12b8f/fbgemm_gpu/fbgemm_gpu/split_table_batched_embeddings_ops.py#L676-L678
+
+    # Note that lr_decay, weight_decay and initial_accumulator_value for Adagrad optimizer in FBGEMM v0.3.2
+    # cannot be specified below. This equivalently means that all these parameters are hardcoded to zero.
+    optimizer_kwargs = {"lr": args.learning_rate}
+    if args.adagrad:
+        optimizer_kwargs["eps"] = args.eps
+    #apply_optimizer_in_backward(
+    #    embedding_optimizer,
+    #    train_model.model.sparse_arch.parameters(),
+    #    optimizer_kwargs,
+    #)
+    #planner = EmbeddingShardingPlanner(
+    #    topology=Topology(
+    #        local_world_size=get_local_size(),
+    #        world_size=dist.get_world_size(),
+    #        compute_device=device.type,
+    #    ),
+    #    batch_size=args.batch_size,
+    #    # If experience OOM, increase the percentage. see
+    #    # https://pytorch.org/torchrec/torchrec.distributed.planner.html#torchrec.distributed.planner.storage_reservations.HeuristicalStorageReservation
+    #    storage_reservation=HeuristicalStorageReservation(percentage=0.05),
+    #)
+    #plan = planner.collective_plan(
+    #    train_model, get_default_sharders(), dist.GroupMember.WORLD
+    #)
+
+    #model = DistributedModelParallel(
+    #    module=train_model,
+    #    device=device,
+    #    plan=plan,
+    #)
+    #if rank == 0 and args.print_sharding_plan:
+    #    for collectionkey, plans in model._plan.plan.items():
+    #        print(collectionkey)
+    #        for table_name, plan in plans.items():
+    #            print(table_name, "\n", plan, "\n")
+
+    model = train_model
+    model.model.sparse_arch = SparseArchTraceAbleWrapper(model.model.sparse_arch)
+
+
+    if ext_dist.my_size > 1:
+        if args.use_device == "cuda":
+            device_ids = [rank]
+            #print(device_ids)
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+        elif args.use_device == "xpu":
+            #device_ids = ["xpu:{}".format(ext_dist.my_local_rank)]
+            device_ids = [device]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+        else:
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch)
+
+    print(model)
+    #def optimizer_with_params():
+    #    if args.adagrad:
+    #        return lambda params: torch.optim.Adagrad(params, lr=args.learning_rate, eps=args.eps)
+    #    else:
+    #        return lambda params: torch.optim.SGD(params, lr=args.learning_rate)
+
+    #optimizer = KeyedOptimizerWrapper(
+    #    dict(model.named_parameters()),
+    #    optimizer_with_params(),
+    #)
+    #for k, v in optimizer.state_dict()['state'].items():
+    #    print(k)
+    #    print(v['step'].device)
+    #    print(v['sum'].device)
+    #exit()
+    #dense_optimizer = KeyedOptimizerWrapper(
+    #    dict(in_backward_optimizer_filter(model.named_parameters())),
+    #    optimizer_with_params(),
+    #)
+    #optimizer = CombinedOptimizer([model.fused_optimizer, dense_optimizer])
+    #optimizer = CombinedOptimizer([optimizer])
+    optimizer = None
+    lr_scheduler = None
+    if not args.inference_only:
+        if args.adagrad:
+            optimizer = torch.optim.Adagrad(model.parameters(), lr=args.learning_rate, eps=args.eps)
+        else:
+            optimizer = torch.optim.SGD(model.parameters(), lr=args.learning_rate)
+        lr_scheduler = LRPolicyScheduler(
+            optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
+        )
+
+    if args.use_device == "xpu":
+        if not args.inference_only:
+            model.train()
+            model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                      dtype=torch.float32)
+        else:
+            model.eval()
+            model = torch.xpu.optimize(model=model, dtype=args.amp_dtype)
+
+    if args.trans_data and (args.multi_hot_sizes is not None):
+        multihot = Multihot(
+            args.multi_hot_sizes,
+            args.num_embeddings_per_feature,
+            args.batch_size,
+            collect_freqs_stats=args.collect_multi_hot_freqs_stats,
+            dist_type=args.multi_hot_distribution_type,
+        )
+        multihot.pause_stats_collection_during_val_and_test(model)
+        train_dataloader = RestartableMap(
+            multihot.convert_to_multi_hot, train_dataloader
+        )
+        val_dataloader = RestartableMap(multihot.convert_to_multi_hot, val_dataloader)
+        test_dataloader = RestartableMap(multihot.convert_to_multi_hot, test_dataloader)
+    train_val_test(
+        args,
+        model,
+        optimizer,
+        device,
+        train_dataloader,
+        val_dataloader,
+        test_dataloader,
+        lr_scheduler,
+    )
+    if args.collect_multi_hot_freqs_stats:
+        multihot.save_freqs_stats()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py
new file mode 100644
index 000000000..865eba420
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py
@@ -0,0 +1,663 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import builtins
+import os
+import sys
+
+import torch
+import torch.distributed as dist
+from torch.autograd import Function
+from torch.autograd.profiler import record_function
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+try:
+    import intel_extension_for_pytorch
+except ImportError as e:
+    pass
+
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+        torch_ccl = False
+    else:
+        import torch_ccl
+except ImportError as e:
+    # print(e)
+    torch_ccl = False
+
+try:
+    import torch_ucc
+except ImportError as e:
+    torch_ucc = False
+
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+a2a_impl = os.environ.get("DLRM_ALLTOALL_IMPL", "")
+
+myreq = None
+
+
+def env2int(env_list, default=-1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0:
+            return val
+    return default
+
+
+def get_my_slice(n):
+    tmp_size = dist.get_world_size()
+    tmp_rank = dist.get_rank()
+
+    k, m = divmod(n, tmp_size)
+    return slice(
+        tmp_rank * k + min(tmp_rank, m), (tmp_rank + 1) * k + min(tmp_rank + 1, m), 1
+    )
+
+
+def get_split_lengths(n):
+    if my_size >1:
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+    else:
+        tmp_size = my_size
+        tmp_rank = my_rank
+
+    k, m = divmod(n, tmp_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        splits = [(k + 1) if i < m else k for i in range(tmp_size)]
+        my_len = splits[tmp_rank]
+    return (my_len, splits)
+
+
+def init_distributed(rank=-1, local_rank=-1, size=-1, use_device="cpu", backend=""):
+    global myreq
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(
+        ["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_SIZE", "WORLD_SIZE"]
+    )
+    if backend == "" and num_mpi_ranks > 1:
+        if torch_ccl or oneccl_bindings_for_pytorch:
+            backend = "ccl"
+        elif use_device=="cuda" and dist.is_nccl_available():
+            backend = "nccl"
+        elif dist.is_mpi_available():
+            backend = "mpi"
+        else:
+            print(
+                "WARNING: MPI multi-process launch detected but PyTorch MPI backend not available."
+            )
+            backend = "gloo"
+
+    if backend != "":
+        # guess Rank and size
+        if rank == -1:
+            rank = env2int(
+                ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+            )
+        if size == -1:
+            size = env2int(
+                [
+                    "PMI_SIZE",
+                    "OMPI_COMM_WORLD_SIZE",
+                    "MV2_COMM_WORLD_SIZE",
+                    "WORLD_SIZE",
+                ],
+                1,
+            )
+        if not os.environ.get("RANK", None) and rank != -1:
+            os.environ["RANK"] = str(rank)
+        if not os.environ.get("WORLD_SIZE", None) and size != -1:
+            os.environ["WORLD_SIZE"] = str(size)
+        if not os.environ.get("MASTER_PORT", None):
+            os.environ["MASTER_PORT"] = "29500"
+        if not os.environ.get("MASTER_ADDR", None):
+            local_size = env2int(
+                [
+                    "MPI_LOCALNRANKS",
+                    "OMPI_COMM_WORLD_LOCAL_SIZE",
+                    "MV2_COMM_WORLD_LOCAL_SIZE",
+                ],
+                1,
+            )
+            if local_size != size and backend != "mpi":
+                print(
+                    "Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default"
+                )
+                print(
+                    "If this run hangs, try exporting rank 0's hostname as MASTER_ADDR"
+                )
+            os.environ["MASTER_ADDR"] = "127.0.0.1"
+
+    if size > 1:
+        if local_rank == -1:
+            my_local_rank = env2int(
+                [
+                    "MPI_LOCALRANKID",
+                    "OMPI_COMM_WORLD_LOCAL_RANK",
+                    "MV2_COMM_WORLD_LOCAL_RANK",
+                    "LOCAL_RANK",
+                ],
+                0,
+            )
+        else:
+            my_local_rank = local_rank
+        my_local_size = env2int(
+            [
+                "MPI_LOCALNRANKS",
+                "OMPI_COMM_WORLD_LOCAL_SIZE",
+                "MV2_COMM_WORLD_LOCAL_SIZE",
+            ],
+            1,
+        )
+        if use_device=="cuda":
+            if my_local_size > torch.cuda.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.cuda.device_count())
+                )
+                sys.exit(1)
+            torch.cuda.set_device(my_local_rank)
+        elif use_device=="xpu":
+            if my_local_size > torch.xpu.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.xpu.device_count())
+                )
+                sys.exit(1)
+            torch.xpu.set_device(my_local_rank)
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        this_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        #my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        #my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if this_rank == 0:
+            print("Running on %d ranks using %s backend" % (my_size, backend))
+        if hasattr(dist, "all_to_all_single"):
+            try:
+                if use_device == "xpu":
+                    test_device: torch.device = torch.device("xpu", dist.get_rank())
+                elif use_device == "cuda":
+                    test_device: torch.device = torch.device(f"cuda:{dist.get_rank()}")
+                else:
+                    pass
+
+                a = torch.rand([1, 3]).to(test_device)
+                dist.broadcast(a, src=0)
+                print("Successfully execute distributed broadcast on the xpu:{}".format(this_rank))
+
+                tensor_list = [torch.zeros(2, dtype=torch.int64).to(test_device) for _ in range(my_size)]
+                tensor = (torch.arange(2, dtype=torch.int64) + 1 + 2 * rank).to(test_device)
+                dist.all_gather(tensor_list, tensor)
+                print("Successfully execute distributed all_gather on the xpu:{}".format(this_rank))
+
+                t = torch.zeros([4*my_size])
+                if use_device == "cuda":
+                    t = t.cuda()
+                elif  use_device == "xpu":
+                    t = t.xpu()
+                dist.all_to_all_single(t, t)
+
+                print("Successfully execute distributed all_to_all_single on the xpu:{}".format(this_rank))
+                alltoall_supported = True
+            except RuntimeError as err:
+                print("fail to enable all_to_all_single primitive: %s" % err)
+                exit()
+        if a2a_impl == "alltoall" and alltoall_supported == False:
+            print(
+                "Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall"
+                % (a2a_impl, backend)
+            )
+            a2a_impl = "scatter"
+        if a2a_impl != "":
+            print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+    else:
+        this_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    print_all(
+        "world size: %d, current rank: %d, local rank: %d"
+        % (my_size, this_rank, my_local_rank)
+    )
+    myreq = Request()
+
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            for j in range(table_split_lengths[i]):
+                out_tensor = inputs[0].new_empty(
+                    [a2a_info.local_batch_num, a2a_info.emb_dim]
+                )
+                scatter_list = (
+                    list(inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else []
+                )
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        a2a_info = ctx.a2a_info
+        grad_output = [t.contiguous() for t in grad_output]
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else [a2a_info.local_batch_num] * tmp_size
+        )
+        per_rank_table_splits = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_inputs = [
+            grad_output[0].new_empty([ctx.a2a_info.batch_size, ctx.a2a_info.emb_dim])
+            for _ in range(a2a_info.local_table_num)
+        ]
+        req_list = []
+        ind = 0
+        for i in range(tmp_size):
+            for j in range(per_rank_table_splits[i]):
+                gather_list = (
+                    list(grad_inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else None
+                )
+                req = dist.gather(grad_output[ind], gather_list, dst=i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(batch_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            out_tensor = input.new_empty(
+                [a2a_info.local_batch_num, table_split_lengths[i] * a2a_info.emb_dim]
+            )
+            req = dist.scatter(
+                out_tensor, scatter_list if i == tmp_rank else [], src=i, async_op=True
+            )
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        ctx.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2a_info.emb_dim, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        assert len(grad_output) == tmp_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2a_info = ctx.a2a_info
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_input = grad_output[0].new_empty(
+            [a2a_info.batch_size, a2a_info.emb_dim * a2a_info.local_table_num]
+        )
+        gather_list = list(grad_input.split(batch_split_lengths, dim=0))
+        req_list = []
+        for i in range(tmp_size):
+            req = dist.gather(
+                scatter_list[i],
+                gather_list if i == tmp_rank else [],
+                dst=i,
+                async_op=True,
+            )
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        with record_function("DLRM alltoall_req_fwd_single"):
+            batch_split_lengths = a2a_info.global_batch_partition_slices
+            if batch_split_lengths:
+                batch_split_lengths = [
+                    m * a2a_info.emb_dim * a2a_info.local_table_num
+                    for m in batch_split_lengths
+                ]
+            table_split_lengths = a2a_info.global_table_wise_parition_slices
+            if table_split_lengths:
+                table_split_lengths = [
+                    a2a_info.local_batch_num * e * a2a_info.emb_dim
+                    for e in table_split_lengths
+                ]
+            input = torch.cat(inputs, dim=1).view([-1])
+            output = input.new_empty(
+                [
+                    a2a_info.global_table_num
+                    * a2a_info.local_batch_num
+                    * a2a_info.emb_dim
+                ]
+            )
+            req = dist.all_to_all_single(
+                output, input, table_split_lengths, batch_split_lengths, async_op=True
+            )
+
+            myreq.req = req
+            myreq.tensor = []
+            myreq.tensor.append(output)
+            myreq.tensor = tuple(myreq.tensor)
+            a2a_info.batch_split_lengths = batch_split_lengths
+            a2a_info.table_split_lengths = table_split_lengths
+            myreq.a2a_info = a2a_info
+            ctx.a2a_info = a2a_info
+            return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        with record_function("DLRM alltoall_req_bwd_single"):
+            a2a_info = ctx.a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            grad_input = myreq.tensor
+            grad_inputs = grad_input.view([a2a_info.batch_size, -1]).split(
+                a2a_info.emb_dim, dim=1
+            )
+            grad_inputs = [gin.contiguous() for gin in grad_inputs]
+            myreq.tensor = None
+            return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        with record_function("DLRM alltoall_wait_fwd_single"):
+            a2a_info = myreq.a2a_info
+            ctx.a2a_info = a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            myreq.tensor = None
+            table_split_lengths = (
+                a2a_info.table_split_lengths
+                if a2a_info.table_split_lengths
+                else a2a_info.local_table_num
+                * a2a_info.local_batch_num
+                * a2a_info.emb_dim
+            )
+            outputs = output[0].split(table_split_lengths)
+            outputs = tuple(
+                [out.view([a2a_info.local_batch_num, -1]) for out in outputs]
+            )
+            return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        with record_function("DLRM alltoall_wait_bwd_single"):
+            a2a_info = ctx.a2a_info
+            grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+            grad_output = torch.cat(grad_outputs)
+            grad_input = grad_output.new_empty(
+                [a2a_info.batch_size * a2a_info.local_table_num * a2a_info.emb_dim]
+            )
+            req = dist.all_to_all_single(
+                grad_input,
+                grad_output,
+                a2a_info.batch_split_lengths,
+                a2a_info.table_split_lengths,
+                async_op=True,
+            )
+            myreq.req = req
+            myreq.tensor = grad_input
+            return (grad_output,)
+
+
+class AllGather(Function):
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * tmp_size
+
+        assert len(global_lengths) == tmp_size
+        assert global_lengths[tmp_rank] == input.size(dim)
+        local_start = sum(global_lengths[:tmp_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[tmp_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(tmp_size)]
+            gather_list = []
+            for length in global_lengths:
+                output_size[dim] = length
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+
+class All2AllInfo(object):
+    pass
+
+
+def alltoall(inputs, per_rank_table_splits):
+    tmp_size = dist.get_world_size()
+    global myreq
+    batch_size, emb_dim = inputs[0].size()
+    a2a_info = All2AllInfo()
+    a2a_info.local_table_num = len(inputs)
+    a2a_info.global_table_wise_parition_slices = per_rank_table_splits
+    (
+        a2a_info.local_batch_num,
+        a2a_info.global_batch_partition_slices,
+    ) = get_split_lengths(batch_size)
+    a2a_info.emb_dim = emb_dim
+    a2a_info.batch_size = batch_size
+    a2a_info.global_table_num = (
+        sum(per_rank_table_splits)
+        if per_rank_table_splits
+        else a2a_info.local_table_num * tmp_size
+    )
+
+    if a2a_impl == "" and alltoall_supported or a2a_impl == "alltoall":
+        # print("Using All2All_Req")
+        output = All2All_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == "" or a2a_impl == "scatter":
+        # print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == "scatter_list":
+        # print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print(
+            "Unknown value set for DLRM_ALLTOALL_IMPL (%s), "
+            "please use one of [alltoall, scatter, scatter_list]" % a2a_impl
+        )
+    return myreq
+
+
+def all_gather(input, lengths, dim=0):
+    tmp_size = dist.get_world_size()
+    if not lengths:
+        lengths = [input.size(0)] * tmp_size
+    return AllGather.apply(input, lengths, dim)
+
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
+
+# Override builtin print function to print only from rank 0
+orig_print = builtins.print
+#
+#
+#def rank0_print(*args, **kwargs):
+#    if my_rank <= 0 or kwargs.get("print_all", False):
+#        orig_print(*args, **kwargs)
+#
+#
+#builtins.print = rank0_print
+#
+# Allow printing from all rank with explicit print_all
+def print_all(*args, **kwargs):
+    orig_print(*args, **kwargs)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py
new file mode 100644
index 000000000..673111dc0
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py
@@ -0,0 +1,67 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+from torchrec.models.dlrm import SparseArch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from typing import List
+
+def unpack(input :KeyedJaggedTensor, default_embedding_names:[str]) -> dict:
+    output = {}
+    for k, v in input.to_dict().items():
+        if k in default_embedding_names:
+            output[k] = {}
+            output[k]['values'] = v._values.to(torch.int64)
+            output[k]['offsets'] = v._offsets.to(torch.int64)
+    return output
+
+
+class SparseArchTraceAbleWrapper(nn.Module):
+
+    def __init__(self, sparse_arch: SparseArch) -> None:
+        super().__init__()
+        self.sparse_arch: SparseArch = sparse_arch
+        for embedding_bag in self.sparse_arch.embedding_bag_collection.embedding_bags.values():
+            embedding_bag.sparse = True
+
+    def forward(
+        self,
+        features: dict,
+    ) -> torch.Tensor:
+        """
+        Args:
+            features (dict): input tensors of sparse features.
+        Returns:
+            torch.Tensor: tensor of shape B X F X D.
+        """
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        pooled_embeddings: List[torch.Tensor] = []
+        for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+            for feature_name in embedding_bag_collection._feature_names[i]:
+                f = features[feature_name]
+                res = embedding_bag(
+                    f['values'],
+                    f['offsets'],
+                    per_sample_weights=None,
+                )
+                pooled_embeddings.append(res)
+        #data = torch.cat(pooled_embeddings, dim=1)
+        #B: int = features['cat_0']['offsets'].numel() - 1
+        #return data.reshape(B, self.sparse_arch.F, self.sparse_arch.D)
+        return pooled_embeddings
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py
new file mode 100644
index 000000000..4aebce416
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Copied from https://github.com/facebookresearch/dlrm/blob/mlperf/dlrm_s_pytorch.py
+
+import sys
+
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py
new file mode 100644
index 000000000..77b13b9e8
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py
@@ -0,0 +1,175 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import List, Tuple
+
+import numpy as np
+
+import torch
+from torchrec.datasets.utils import Batch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class RestartableMap:
+    def __init__(self, f, source):
+        self.source = source
+        self.func = f
+
+    def __iter__(self):
+        for x in self.source:
+            yield self.func(x)
+
+    def __len__(self):
+        return len(self.source)
+
+
+class Multihot:
+    def __init__(
+        self,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+        collect_freqs_stats: bool,
+        dist_type: str = "uniform",
+    ):
+        if dist_type not in {"uniform", "pareto"}:
+            raise ValueError(
+                "Multi-hot distribution type {} is not supported."
+                'Only "uniform" and "pareto" are supported.'.format(dist_type)
+            )
+        self.dist_type = dist_type
+        self.multi_hot_sizes = multi_hot_sizes
+        self.num_embeddings_per_feature = num_embeddings_per_feature
+        self.batch_size = batch_size
+
+        # Generate 1-hot to multi-hot lookup tables, one lookup table per sparse embedding table.
+        self.multi_hot_tables_l = self.__make_multi_hot_indices_tables(
+            dist_type, multi_hot_sizes, num_embeddings_per_feature
+        )
+
+        # Pooling offsets are computed once and reused.
+        self.offsets = self.__make_offsets(
+            multi_hot_sizes, num_embeddings_per_feature, batch_size
+        )
+
+        # For plotting frequency access
+        self.collect_freqs_stats = collect_freqs_stats
+        self.model_to_track = None
+        self.freqs_pre_hash = []
+        self.freqs_post_hash = []
+        for embs_count in num_embeddings_per_feature:
+            self.freqs_pre_hash.append(np.zeros((embs_count)))
+            self.freqs_post_hash.append(np.zeros((embs_count)))
+
+    def save_freqs_stats(self) -> None:
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            rank = torch.distributed.get_rank()
+        else:
+            rank = 0
+        pre_dict = {str(k): e for k, e in enumerate(self.freqs_pre_hash)}
+        np.save(f"stats_pre_hash_{rank}_{self.dist_type}.npy", pre_dict)
+        post_dict = {str(k): e for k, e in enumerate(self.freqs_post_hash)}
+        np.save(f"stats_post_hash_{rank}_{self.dist_type}.npy", post_dict)
+
+    def pause_stats_collection_during_val_and_test(
+        self, model: torch.nn.Module
+    ) -> None:
+        self.model_to_track = model
+
+    def __make_multi_hot_indices_tables(
+        self,
+        dist_type: str,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+    ) -> List[np.array]:
+        np.random.seed(
+            0
+        )  # The seed is necessary for all ranks to produce the same lookup values.
+        multi_hot_tables_l = []
+        for embs_count, multi_hot_size in zip(
+            num_embeddings_per_feature, multi_hot_sizes
+        ):
+            embedding_ids = np.arange(embs_count)[:, np.newaxis]
+            if dist_type == "uniform":
+                synthetic_sparse_ids = np.random.randint(
+                    0, embs_count, size=(embs_count, multi_hot_size - 1)
+                )
+            elif dist_type == "pareto":
+                synthetic_sparse_ids = (
+                    np.random.pareto(
+                        a=0.25, size=(embs_count, multi_hot_size - 1)
+                    ).astype(np.int32)
+                    % embs_count
+                )
+            multi_hot_table = np.concatenate(
+                (embedding_ids, synthetic_sparse_ids), axis=-1
+            )
+            multi_hot_tables_l.append(multi_hot_table)
+        multi_hot_tables_l = [
+            torch.from_numpy(multi_hot_table).int()
+            for multi_hot_table in multi_hot_tables_l
+        ]
+        return multi_hot_tables_l
+
+    def __make_offsets(
+        self,
+        multi_hot_sizes: int,
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+    ) -> List[torch.Tensor]:
+        lS_o = torch.ones(
+            (len(num_embeddings_per_feature) * batch_size), dtype=torch.int32
+        )
+        for k, multi_hot_size in enumerate(multi_hot_sizes):
+            lS_o[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        lS_o = torch.cumsum(torch.concat((torch.tensor([0]), lS_o)), axis=0)
+        return lS_o
+
+    def __make_new_batch(
+        self,
+        lS_i: torch.Tensor,
+        batch_size: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        lS_i = lS_i.reshape(-1, batch_size)
+        multi_hot_ids_l = []
+        for k, (sparse_data_batch_for_table, multi_hot_table) in enumerate(
+            zip(lS_i, self.multi_hot_tables_l)
+        ):
+            multi_hot_ids = torch.nn.functional.embedding(
+                sparse_data_batch_for_table, multi_hot_table
+            )
+            multi_hot_ids = multi_hot_ids.reshape(-1)
+            multi_hot_ids_l.append(multi_hot_ids)
+            if self.collect_freqs_stats and (
+                self.model_to_track is None or self.model_to_track.training
+            ):
+                idx_pre, cnt_pre = np.unique(
+                    sparse_data_batch_for_table, return_counts=True
+                )
+                idx_post, cnt_post = np.unique(multi_hot_ids, return_counts=True)
+                self.freqs_pre_hash[k][idx_pre] += cnt_pre
+                self.freqs_post_hash[k][idx_post] += cnt_post
+        lS_i = torch.cat(multi_hot_ids_l)
+        if batch_size == self.batch_size:
+            return lS_i, self.offsets
+        else:
+            return lS_i, self.__make_offsets(
+                self.multi_hot_sizes, self.num_embeddings_per_feature, batch_size
+            )
+
+    def convert_to_multi_hot(self, batch: Batch) -> Batch:
+        batch_size = len(batch.dense_features)
+        lS_i = batch.sparse_features._values
+        lS_i, lS_o = self.__make_new_batch(lS_i, batch_size)
+        new_sparse_features = KeyedJaggedTensor.from_offsets_sync(
+            keys=batch.sparse_features._keys,
+            values=lS_i,
+            offsets=lS_o,
+        )
+        return Batch(
+            dense_features=batch.dense_features,
+            sparse_features=new_sparse_features,
+            labels=batch.labels,
+        )
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/requirements.txt b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/requirements.txt
new file mode 100644
index 000000000..270005d8d
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/requirements.txt
@@ -0,0 +1,4 @@
+fbgemm-gpu==0.3.2
+torchmetrics==0.11.0
+torchrec==0.3.2
+torchsnapshot
diff --git a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/__init__.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py
similarity index 72%
rename from models/image_recognition/tensorflow/resnet50v1_5/inference/int8/__init__.py
rename to models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py
index 159180624..1b67e31e9 100644
--- a/models/image_recognition/tensorflow/resnet50v1_5/inference/int8/__init__.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py
@@ -1,20 +1,16 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
+# Copyright 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#    http://www.apache.org/licenses/LICENSE-2.0
+#     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
 
+from .generate_plan import generate_plan
+from .planner import Cost, CostModel, Planner, ShardingState
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py
new file mode 100644
index 000000000..8c328cbac
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from argparse import Namespace
+from itertools import chain
+from typing import List
+
+from .planner import CostModel, Planner
+
+
+def generate_plan(
+    slot_size_array: List[int],
+    multi_hot_sizes: List[int],
+    num_nodes: int,
+    num_gpus: int,
+    args: Namespace,
+    log_result: bool,
+):
+    def sanity_check(shard_matrix, shard_strategy):
+        # mainly to make sure all the tables are sharded
+        msg = "Not all tables covered in the sharding plan"
+        assert set(chain(*shard_matrix)) == set(range(len(slot_size_array))), msg
+        shard_strategy_list = [x for strategy_pair in shard_strategy for x in strategy_pair[1]]
+        assert set(shard_strategy_list) == set(range(len(slot_size_array))), msg
+
+        for table_list in shard_matrix:
+            if len(table_list) == 0:
+                raise Exception("Currently no empty shard list is allowed")
+
+    def int_to_string(shard_matrix_int, shard_strategy_int):
+        shard_strategy, shard_matrix = [], []
+        for pair in shard_strategy_int:
+            if len(pair[1]) != 0:
+                shard_strategy.append((pair[0], [str(x) for x in pair[1]]))
+        for sub_matrix_ in shard_matrix_int:
+            shard_matrix.append([str(x) for x in sub_matrix_])
+        return shard_matrix, shard_strategy
+
+    if args.sharding_plan in ["round_robin", "uniform"]:
+        # sharding strategies that don't exploit system configs
+        if args.sharding_plan == "round_robin":
+            mp_table = [i for i in range(len(slot_size_array))]
+            shard_matrix_ = [[] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in mp_table])]
+
+            for i, table_id in enumerate(mp_table):
+                target_gpu = i % num_gpus
+                shard_matrix_[target_gpu].append(table_id)
+
+        elif args.sharding_plan == "uniform":
+            shard_matrix_ = [[x for x in range(len(slot_size_array))] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in range(len(slot_size_array))])]
+
+    elif args.sharding_plan in ["auto", "hier_auto"]:
+        # sharding strategies that exploit system configs
+        dram_cap = args.memory_cap_for_embedding
+        if args.optimizer == "adagrad":
+            byte_per_elem = 8
+        elif args.optimizer == "sgd":
+            byte_per_elem = 4
+
+        if args.sharding_plan == "auto":
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_gpus, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_ = planner.plan()
+
+        elif args.sharding_plan == "hier_auto":
+            if num_nodes <= 1:
+                raise Exception("hier_auto plan is only applicable to configs with more than one node")
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap * args.num_gpus_per_node,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_nodes, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_node_ = planner.plan()
+            shard_matrix_ = []
+            for node_shard_matrix in shard_matrix_node_:
+                for i in range(args.num_gpus_per_node):
+                    shard_matrix_.append(node_shard_matrix)
+    elif args.sharding_plan in ["max_min"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[] for _ in range(num_gpus)]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+        dict_feature = {}
+        for k, v in enumerate(slot_size_array):
+            dict_feature[k] = v
+        dict_feature_list = sorted(dict_feature.items(),key=lambda x:x[1], reverse=True)
+        end = len(dict_feature_list) - 1
+        for i in range(len(dict_feature_list)):
+            index = i%num_gpus if int(i/num_gpus) == 0 else (num_gpus -1 - i%num_gpus)
+            shard_matrix_[index].append(dict_feature_list[i][0])
+            if (i >= end):
+                break
+            elif(end - i >= num_gpus - i%num_gpus):
+                shard_matrix_[index].append(dict_feature_list[end][0])
+                end -= 1
+    elif args.sharding_plan in ["custom"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[0, 5], [9, 16], [19, 12], [20, 18], [21, 25], [10, 8, 2, 13, 3], [22, 24, 4, 7, 23], [11, 15, 1, 17, 14, 6]]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+    else:
+        raise Exception("unknown sharding plan")
+
+    sanity_check(shard_matrix_, shard_strategy_)
+    shard_matrix, shard_strategy = int_to_string(shard_matrix_, shard_strategy_)
+
+    if log_result:
+        logging.info("Provided system info: ")
+        #logging.info("num_gpu_per_nodes: %d", args.num_gpus_per_node)
+        #logging.info("Memory to communication BW ratio: %f", args.mem_comm_bw_ratio)
+        #logging.info("Memory to communication work ratio: %f", args.mem_comm_work_ratio)
+        #logging.info("DRAM capacity: %f GB", args.memory_cap_for_embedding)
+        logging.info("shard_matrix:")
+        logging.info(shard_matrix)
+        logging.info("\n")
+
+    return shard_matrix, shard_strategy, shard_matrix_, shard_strategy_
diff --git a/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py
new file mode 100644
index 000000000..c521ac392
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py
@@ -0,0 +1,314 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import time
+from typing import List, Tuple
+
+import numpy as np
+
+
+class ShardingState:
+    """
+    Containing the state of a sharding process.
+    The plan iteratively update the sharding state based on a given heuristic and obtain
+    solutions.
+    """
+
+    def __init__(
+        self,
+        array_hotness: np.array,
+        num_bucket: int,
+        dp_table_id: np.array(int) = np.array([]),
+    ) -> None:
+        mp_table_id = np.setdiff1d(np.arange(array_hotness.size), dp_table_id)
+        array_hotness_mp = array_hotness[mp_table_id]
+        sorted_idx = np.argsort(array_hotness_mp)[::-1]
+        self.array_unshard_hotness = array_hotness
+        self.array_hotness = array_hotness_mp[sorted_idx]
+        self.num_bucket = num_bucket
+        self.array_table_id = mp_table_id[sorted_idx]
+        self.array_num_split = np.zeros(self.array_unshard_hotness.size, dtype=int)
+        self.array_num_split[mp_table_id] = 1
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def split_hot_shard(self):
+        """
+        split the shard with the largest hotness
+        """
+        # shards are sorted based on the hotness. Find the first hot shard that
+        # can be split further
+        for shard_id in range(self.array_table_id.size):
+            table_id = self.array_table_id[shard_id]
+            hotness = self.array_unshard_hotness[table_id]
+            if self.array_num_split[table_id] * 2 <= self.num_bucket:
+                # if this table can be furter splitted and we can put it into
+                # more buckets
+                idx = np.where(self.array_table_id == table_id)[0]
+                self.array_hotness = np.delete(self.array_hotness, idx)
+                self.array_table_id = np.delete(self.array_table_id, idx)
+                self.array_num_split[table_id] *= 2
+                self.array_hotness = np.concatenate(
+                    (
+                        self.array_hotness,
+                        np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                    )
+                )
+                self.array_table_id = np.concatenate(
+                    (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+                )
+                break
+
+        # sort after splitting to maintain the shard hotness in order
+        sorted_idx = np.argsort(self.array_hotness)[::-1]
+        self.array_hotness = self.array_hotness[sorted_idx]
+        self.array_table_id = self.array_table_id[sorted_idx]
+
+    def split_oom_shard(self, table_id: int) -> bool:
+        hotness = self.array_unshard_hotness[table_id]
+        if self.array_num_split[table_id] * 2 <= self.num_bucket:
+            idx = np.where(self.array_table_id == table_id)[0]
+            self.array_hotness = np.delete(self.array_hotness, idx)
+            self.array_table_id = np.delete(self.array_table_id, idx)
+            self.array_num_split[table_id] *= 2
+            self.array_hotness = np.concatenate(
+                (
+                    self.array_hotness,
+                    np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                )
+            )
+            self.array_table_id = np.concatenate(
+                (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+            )
+            sorted_idx = np.argsort(self.array_hotness)[::-1]
+            self.array_hotness = self.array_hotness[sorted_idx]
+            self.array_table_id = self.array_table_id[sorted_idx]
+            return True
+        else:
+            return False
+
+    def update_split_num(self):
+        self.array_num_split = np.zeros_like(self.array_unshard_hotness)
+        for shard_list in self.shard_ll:
+            for table_id in shard_list:
+                self.array_num_split[table_id] += 1
+
+    def reset_shard_ll(self):
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def push_bucket(
+        self,
+        bucket_id: int,
+        table_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].append(table_id)
+
+    def pop_bucket(
+        self,
+        bucket_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].pop()
+
+
+class Cost:
+    def __init__(
+        self,
+        cost: np.array(float),
+        hotness_cost: np.array(float),
+        table_cost: np.array(float),
+        mem_cost: np.array(float),
+    ) -> None:
+        self.cost = cost
+        self.hotness_cost = hotness_cost
+        self.table_cost = table_cost
+        self.mem_cost = mem_cost
+
+
+class CostModel:
+    def __init__(
+        self,
+        hotness_cost: float,
+        table_cost: float,
+        mem_cost: float,
+        mem_capacity: float,
+        table_size: List[int],
+    ) -> None:
+        self.unit_hotness_cost = hotness_cost
+        self.unit_table_cost = table_cost
+        self.unit_mem_cost = mem_cost
+        self.mem_capacity = mem_capacity
+        self.array_table_size = np.array(table_size)
+
+    def get_cost(
+        self,
+        ss: ShardingState,
+    ) -> Tuple[Cost, bool]:
+        list_cost = []
+        list_hotness_cost = []
+        list_table_cost = []
+        list_mem_cost = []
+
+        for shard_list in ss.shard_ll:
+            hotness_cost = (
+                self.unit_hotness_cost
+                * (ss.array_unshard_hotness[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            table_cost = self.unit_table_cost * len(shard_list)
+            mem_cost = (
+                self.unit_mem_cost
+                * (self.array_table_size[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            list_cost.append(hotness_cost + table_cost)
+            list_hotness_cost.append(hotness_cost)
+            list_table_cost.append(table_cost)
+            list_mem_cost.append(mem_cost)
+
+        return (
+            Cost(np.array(list_cost), np.array(list_hotness_cost), np.array(list_table_cost), np.array(list_mem_cost)),
+            max(list_mem_cost) > self.mem_capacity,
+        )
+
+    def deduct_mem_cap_for_dp(
+        self,
+        dp_table_id: list,
+    ) -> None:
+        self.mem_capacity -= self.array_table_size[dp_table_id].sum() * self.unit_mem_cost
+        if self.mem_capacity < 0:
+            raise Exception("OOM due to DP. Please considering increase the DP threshold")
+
+
+class Planner:
+    """
+    The planner work out a series of plans iteratively.
+    In each iteration, the planner tries to split the hottest shard and place the shards into
+    a bucket based on a give heuristic. When the shard is too large to fit into the best bucket
+    suggested by the heuristic, it finds the next best bucket until it iterates through all the
+    buckets. In that case, it tries to split the shard further. If the shard cannot be split
+    further, the planner aborts and returns the default sharding plan.
+    """
+
+    def __init__(
+        self,
+        list_hotness: list,
+        num_bucket: int,
+        cost_model: CostModel,
+        dp_threshold: int = 0,
+        max_search_iter: int = 20,
+        log_result: bool = False,
+    ) -> None:
+        self.array_hotness = np.array(list_hotness)
+        self.num_bucket = num_bucket
+        self.cost_model = cost_model
+        self.list_candidate = []
+        self.max_search_iter = max_search_iter
+        self.log_result = log_result
+
+        # Create the default sharding plan. Throw if even this default sharding plan cannot fit, as
+        # it should be the most memory-efficient
+        sharding_state_default = ShardingState(self.array_hotness, self.num_bucket)
+        for b in range(self.num_bucket):
+            for t in range(self.array_hotness.size):
+                sharding_state_default.push_bucket(b, t)
+        sharding_state_default.update_split_num()
+        cost, oom = self.cost_model.get_cost(sharding_state_default)
+        if oom:
+            raise Exception("OOM even with the most memory-efficient sharding plan")
+        self.list_candidate.append(
+            (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, sharding_state_default.shard_ll)
+        )
+
+        # Create DP sharding plan based on the DP threshold
+        self.dp_table_id = np.where(cost_model.array_table_size < dp_threshold / cost_model.unit_mem_cost)[0]
+        self.mp_table_id = np.setdiff1d(np.arange(self.array_hotness.size), self.dp_table_id)
+        self.sharding_state = ShardingState(self.array_hotness, self.num_bucket, self.dp_table_id)
+        self.cost_model.deduct_mem_cap_for_dp(self.dp_table_id)
+
+        logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    def greedy_plan(self, ss):
+        """
+        This is a heuristic based on greedy policy. The shard is placed to the bucket with the
+        lowest hotness cost
+        """
+        array_cost = np.zeros(ss.num_bucket)
+        ss.reset_shard_ll()
+        for i in range(ss.array_hotness.size):
+            sorted_idx = np.argsort(array_cost)
+            sharded = False
+            for bucket_id in sorted_idx:
+                if ss.array_table_id[i] not in ss.shard_ll[bucket_id]:
+                    # for now, only uniform sharding is supported. Hence cannot put two shards
+                    # from the same table into the same bucket
+                    ss.push_bucket(bucket_id, ss.array_table_id[i])
+                    cost, oom = self.cost_model.get_cost(ss)
+                    if not oom:
+                        sharded = True
+                        array_cost = cost.cost
+                        break
+                    else:
+                        # Currnet bucket cannot fit. Iterate to the next best bucket
+                        ss.pop_bucket(bucket_id)
+            if not sharded:
+                # This means the shard is too large to fit within any bucket
+                return ss.array_table_id[i], ss, cost
+        return None, ss, cost
+
+    def plan(self):
+        t0 = time.time()
+        for i in range(self.max_search_iter):
+            oom_table_id, self.sharding_state, cost = self.greedy_plan(self.sharding_state)
+            if oom_table_id is None:
+                self.list_candidate.append(
+                    (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, self.sharding_state.shard_ll)
+                )
+                self.sharding_state.split_hot_shard()
+            else:
+                oom_table_can_split = self.sharding_state.split_oom_shard(oom_table_id)
+                if not oom_table_can_split:
+                    break
+
+        self.list_candidate.sort(key=lambda x: x[0])
+
+        shard_strategy = [("mp", self.mp_table_id.tolist())]
+        shard_strategy.append(("dp", self.dp_table_id.tolist()))
+        shard_matrix = self.list_candidate[0][-1]
+        for table_id in self.dp_table_id:
+            for shard_list in shard_matrix:
+                shard_list.append(table_id)
+        if self.log_result:
+            logging.info("Planner took %f sec" % (time.time() - t0))
+            logging.info(shard_strategy)
+            logging.info(shard_matrix)
+            logging.info("hotness cost is:")
+            logging.info(self.list_candidate[0][1])
+            logging.info("table cost is:")
+            logging.info(self.list_candidate[0][2])
+            logging.info("mem cost is:")
+            logging.info(self.list_candidate[0][3])
+        return shard_strategy, shard_matrix
diff --git a/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json b/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
new file mode 100644
index 000000000..01d7d3abb
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
@@ -0,0 +1,33883 @@
+{
+    " ": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 80,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.float32"
+            }
+        ]
+    },
+    "sparse_arch": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'intel_extension_for_pytorch.nn.modules.merged_embeddingbag.MergedEmbeddingBagWithCat'>",
+                "op_type_is_module": true,
+                "fqn": "sparse_arch.embedding_bag_collection",
+                "input_tensor_infos": [
+                    {
+                        "id": 7,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            313710.8125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 8,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            306.2494812011719
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 9,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            135.24298095703125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 10,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            58.20223617553711
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 11,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            158.9283905029297
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 12,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.015685785561800003
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 13,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            55.723880767822266
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 14,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            12.078180313110352
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 15,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.31440040469169617
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 16,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            313715.65625
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 17,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            24061.390625
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 18,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            3178.567626953125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 19,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.07058604061603546
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 20,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            17.317106246948242
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 21,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            93.62061309814453
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 22,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1.2078055143356323
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 23,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.023528680205345154
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 24,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            7.113068580627441
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 25,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.10195761173963547
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 26,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            313715.28125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 27,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            313715.65625
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 28,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            313715.03125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 29,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            4628.4443359375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 30,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            101.72232055664062
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 31,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.45003268122673035
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 32,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            0.26678285002708435
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 33,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            3011.6708984375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 34,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            2007.7806396484375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 35,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 36,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            2007.7806396484375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 37,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            6023.341796875
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 38,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 39,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 40,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 41,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 42,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            7027.232421875
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 43,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            3011.6708984375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 44,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            8031.12255859375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 45,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 46,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            6023.341796875
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 47,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            9035.0126953125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 48,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            5019.45166015625
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 49,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 50,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 51,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 52,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            12046.68359375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 53,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            100389.03125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 54,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            27105.037109375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 55,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            10038.9033203125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 56,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            3011.6708984375
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 57,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 58,
+                        "orig_dtype": "torch.int64",
+                        "inf_dtype": "torch.int64",
+                        "force_dtype": "torch.int64",
+                        "scale": [
+                            1003.8903198242188
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 6,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.00934610515832901
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0007458091131411493
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0017127114115282893
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0014824823010712862
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0007317104609683156
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0006300461827777326
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.005117378663271666
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0011248636292293668
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0017805569805204868
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0025812292005866766
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0008306074305437505
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0013273678487166762
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.000996718998067081
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0030272372532635927
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0011649823281913996
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0008841246599331498
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0012131290277466178
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.003962228540331125
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0009051506640389562
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.002671073190867901
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0007534330943599343
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.000596929807215929
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0006080541643314064
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0015035754768177867
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.001088611432351172
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0027739068027585745
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.002218045061454177
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.03068052977323532
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 59,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.03068052977323532
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "sparse_arch:embedding_bag_collection": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": []
+    },
+    "dense_arch": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 6,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "dense_arch:model": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 6,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "dense_arch:model:_mlp": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 6,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "dense_arch:model:_mlp:0": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "dense_arch.model._mlp.0._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 0,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.1089453399181366
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0022663429845124483,
+                            0.0021414761431515217,
+                            0.0017865091795101762,
+                            0.001973002217710018,
+                            0.0019403076730668545,
+                            0.002047733636572957,
+                            0.0022039003670215607,
+                            0.0022104198578745127,
+                            0.0020569448824971914,
+                            0.0019502975046634674,
+                            0.0022594358306378126,
+                            0.002130665583536029,
+                            0.0022875217255204916,
+                            0.0018153841374441981,
+                            0.00180406856816262,
+                            0.0021904497407376766,
+                            0.002080207224935293,
+                            0.002741212258115411,
+                            0.002520497189834714,
+                            0.0020981812849640846,
+                            0.00182404147926718,
+                            0.002021671738475561,
+                            0.0022021986078470945,
+                            0.002003567526116967,
+                            0.0018605083459988236,
+                            0.002021324820816517,
+                            0.0019739086274057627,
+                            0.002276384737342596,
+                            0.0018277904018759727,
+                            0.0020280107855796814,
+                            0.0018022645963355899,
+                            0.002020078245550394,
+                            0.002239807741716504,
+                            0.0020819753408432007,
+                            0.001916463952511549,
+                            0.002323342952877283,
+                            0.001972865080460906,
+                            0.0020497331861406565,
+                            0.0022463174536824226,
+                            0.0021602814085781574,
+                            0.0022280223201960325,
+                            0.0018567329971119761,
+                            0.0022198280785232782,
+                            0.002005988731980324,
+                            0.002139003947377205,
+                            0.0021627764217555523,
+                            0.001808337983675301,
+                            0.0021895638201385736,
+                            0.0020647873170673847,
+                            0.002608365146443248,
+                            0.0018784789135679603,
+                            0.00226257904432714,
+                            0.0027566642966121435,
+                            0.0021630763076245785,
+                            0.0019869995303452015,
+                            0.001913373707793653,
+                            0.002278856700286269,
+                            0.002237328328192234,
+                            0.0020635563414543867,
+                            0.0021769593004137278,
+                            0.0022780923172831535,
+                            0.0015376710798591375,
+                            0.0020742069464176893,
+                            0.0022271755151450634,
+                            0.002175211673602462,
+                            0.0020885865669697523,
+                            0.0023859748616814613,
+                            0.002209634752944112,
+                            0.0016091886209324002,
+                            0.0017852282617241144,
+                            0.002187066711485386,
+                            0.0019152050372213125,
+                            0.00216384488157928,
+                            0.0023574847728013992,
+                            0.0020712411496788263,
+                            0.0019127834821119905,
+                            0.0019978838972747326,
+                            0.002068493515253067,
+                            0.002058542799204588,
+                            0.0021653713192790747,
+                            0.0019648114684969187,
+                            0.0021225607488304377,
+                            0.0020796179305762053,
+                            0.0021255663596093655,
+                            0.0020920177921652794,
+                            0.002464178018271923,
+                            0.0017674503615126014,
+                            0.0017581740394234657,
+                            0.002277814317494631,
+                            0.0016292465152218938,
+                            0.0020122609566897154,
+                            0.0020672413520514965,
+                            0.002814206760376692,
+                            0.0020466845016926527,
+                            0.00221831863746047,
+                            0.0021372935734689236,
+                            0.0019769740756601095,
+                            0.0021067368797957897,
+                            0.002212358871474862,
+                            0.002290557138621807,
+                            0.0024302946403622627,
+                            0.0024653871078044176,
+                            0.002119719982147217,
+                            0.0021606304217129946,
+                            0.0018107262440025806,
+                            0.0021833195351064205,
+                            0.0020173571538180113,
+                            0.0020492570474743843,
+                            0.001759772771038115,
+                            0.002781154355034232,
+                            0.0019797750283032656,
+                            0.0019158704672008753,
+                            0.0019839578308165073,
+                            0.0021742344833910465,
+                            0.0019714548252522945,
+                            0.0022258127573877573,
+                            0.0017996010137721896,
+                            0.0019882863853126764,
+                            0.0021031787618994713,
+                            0.0020954080391675234,
+                            0.002052443567663431,
+                            0.0021130554378032684,
+                            0.0022030838299542665,
+                            0.002011724282056093,
+                            0.001886627753265202,
+                            0.002112500136718154,
+                            0.0021504699252545834,
+                            0.0016906684031710029,
+                            0.0021892436780035496,
+                            0.0016632371116429567,
+                            0.0020030492451041937,
+                            0.002589726122096181,
+                            0.00241958093829453,
+                            0.0026316724251955748,
+                            0.0016983702080324292,
+                            0.0021774093620479107,
+                            0.001496843178756535,
+                            0.001991849858313799,
+                            0.0021800678223371506,
+                            0.0018424892332404852,
+                            0.0018548744264990091,
+                            0.00212329113855958,
+                            0.001970381010323763,
+                            0.002314050914719701,
+                            0.00221414677798748,
+                            0.002313592005521059,
+                            0.0021945994812995195,
+                            0.0022986005060374737,
+                            0.002198019064962864,
+                            0.002268960466608405,
+                            0.002171639120206237,
+                            0.001952307648025453,
+                            0.002093004761263728,
+                            0.002761349780485034,
+                            0.0023489093873649836,
+                            0.0022109535057097673,
+                            0.0022325757890939713,
+                            0.0021145681384950876,
+                            0.0021614779252558947,
+                            0.0021809893660247326,
+                            0.0021029303316026926,
+                            0.0019338869024068117,
+                            0.001888136612251401,
+                            0.00203246739692986,
+                            0.0020447783172130585,
+                            0.001836742041632533,
+                            0.0021919275168329477,
+                            0.0022366931661963463,
+                            0.00221973261795938,
+                            0.00221843714825809,
+                            0.0021056300029158592,
+                            0.0021744139958173037,
+                            0.002052773954346776,
+                            0.0018757701618596911,
+                            0.0019423309713602066,
+                            0.00197337637655437,
+                            0.0023114504292607307,
+                            0.0019309602212160826,
+                            0.0020318676251918077,
+                            0.002108274959027767,
+                            0.002091656206175685,
+                            0.002083983039483428,
+                            0.0022412093821913004,
+                            0.002090214751660824,
+                            0.0015314753400161862,
+                            0.002083637984469533,
+                            0.002882278524339199,
+                            0.0020392301958054304,
+                            0.002199991140514612,
+                            0.002151302294805646,
+                            0.0021429492626339197,
+                            0.0019335804972797632,
+                            0.0021460263524204493,
+                            0.001983659341931343,
+                            0.002234620274975896,
+                            0.002098308177664876,
+                            0.0019794146064668894,
+                            0.0022068882826715708,
+                            0.0023883841931819916,
+                            0.002137441188097,
+                            0.0020494561176747084,
+                            0.002000269014388323,
+                            0.0016199222300201654,
+                            0.0019931725692003965,
+                            0.0021199050825089216,
+                            0.0019558488857001066,
+                            0.002077266341075301,
+                            0.0022443134803324938,
+                            0.0020825688261538744,
+                            0.0020630343351513147,
+                            0.0021278266794979572,
+                            0.002426191931590438,
+                            0.0017890396993607283,
+                            0.002061880426481366,
+                            0.002542611677199602,
+                            0.002221159404143691,
+                            0.001911692088469863,
+                            0.002027566311880946,
+                            0.0022127379197627306,
+                            0.002281142631545663,
+                            0.002124567050486803,
+                            0.0021506724879145622,
+                            0.0019375201081857085,
+                            0.002084482926875353,
+                            0.0019862737972289324,
+                            0.0020817145705223083,
+                            0.0022463432978838682,
+                            0.0022155006881803274,
+                            0.0021161758340895176,
+                            0.0020477736834436655,
+                            0.0021442940924316645,
+                            0.0019714399240911007,
+                            0.0019705938175320625,
+                            0.002049041446298361,
+                            0.001798365032300353,
+                            0.002129660453647375,
+                            0.0021926965564489365,
+                            0.002433015964925289,
+                            0.002188074868172407,
+                            0.002094882307574153,
+                            0.0018425059970468283,
+                            0.00233240844681859,
+                            0.0021815826185047626,
+                            0.0016963164089247584,
+                            0.0019028132082894444,
+                            0.0023387272376567125,
+                            0.0019090943969786167,
+                            0.002055657561868429,
+                            0.0019647583831101656,
+                            0.0022060961928218603,
+                            0.001888488419353962,
+                            0.002078660996630788,
+                            0.0021024863235652447,
+                            0.0022437828592956066,
+                            0.0021510946098715067,
+                            0.0019881688058376312,
+                            0.0020443303510546684,
+                            0.001980385510250926,
+                            0.0024942264426499605,
+                            0.0021869626361876726,
+                            0.0022781111765652895,
+                            0.002173075918108225,
+                            0.0019246714655309916,
+                            0.0022857061121612787,
+                            0.0019455793080851436,
+                            0.002114281291142106,
+                            0.0019010909600183368,
+                            0.0021440312266349792,
+                            0.001737915794365108,
+                            0.0018988709198310971,
+                            0.0018585219513624907,
+                            0.002226316137239337,
+                            0.0020494526252150536,
+                            0.0015200693160295486,
+                            0.0024201187770813704,
+                            0.002317323349416256,
+                            0.00225347769446671,
+                            0.0023086941801011562,
+                            0.002214839681982994,
+                            0.002218683948740363,
+                            0.0021218329202383757,
+                            0.0022026540245860815,
+                            0.001944970223121345,
+                            0.0020668201614171267,
+                            0.0024896226823329926,
+                            0.0021845854353159666,
+                            0.002159395022317767,
+                            0.0021356185898184776,
+                            0.002015545265749097,
+                            0.0020917453803122044,
+                            0.0022954940795898438,
+                            0.0018914827378466725,
+                            0.001696689985692501,
+                            0.002055762568488717,
+                            0.0023456907365471125,
+                            0.002008291892707348,
+                            0.002051826799288392,
+                            0.002032047603279352,
+                            0.0024981508031487465,
+                            0.0019471977138891816,
+                            0.0021657864563167095,
+                            0.002050461946055293,
+                            0.0016174139454960823,
+                            0.002131626708433032,
+                            0.0020789054688066244,
+                            0.002025837078690529,
+                            0.00205547409132123,
+                            0.001876295660622418,
+                            0.002162245800718665,
+                            0.002094547264277935,
+                            0.0018740310333669186,
+                            0.0021267293486744165,
+                            0.0021177022717893124,
+                            0.002029482973739505,
+                            0.0019914801232516766,
+                            0.002113435883074999,
+                            0.0019828099757432938,
+                            0.0021765774581581354,
+                            0.00201918906532228,
+                            0.0018133067060261965,
+                            0.0019633150659501553,
+                            0.0019890237599611282,
+                            0.001986198592931032,
+                            0.0021675729658454657,
+                            0.0019016485894098878,
+                            0.001964010065421462,
+                            0.0021603009663522243,
+                            0.0021578327286988497,
+                            0.002172536915168166,
+                            0.0025760203134268522,
+                            0.0016426689689978957,
+                            0.00216120108962059,
+                            0.0015783477574586868,
+                            0.0018620279151946306,
+                            0.002635805867612362,
+                            0.002539488486945629,
+                            0.0018467218615114689,
+                            0.0020952618215233088,
+                            0.0015536361606791615,
+                            0.0020399184431880713,
+                            0.0021223928779363632,
+                            0.0020465494599193335,
+                            0.0021071573719382286,
+                            0.0022830914240330458,
+                            0.0017398052150383592,
+                            0.002129546133801341,
+                            0.001978414598852396,
+                            0.0020374085288494825,
+                            0.002119487849995494,
+                            0.0019132365705445409,
+                            0.0020445093978196383,
+                            0.0020999314729124308,
+                            0.0022570211440324783,
+                            0.002780361333861947,
+                            0.0019122208468616009,
+                            0.0024686618708074093,
+                            0.0019565895199775696,
+                            0.0021280706860125065,
+                            0.0021210494451224804,
+                            0.001891457592137158,
+                            0.00221110787242651,
+                            0.002360980724915862,
+                            0.0023327127564698458,
+                            0.001893154694698751,
+                            0.001900819013826549,
+                            0.0021616423036903143,
+                            0.0018140877364203334,
+                            0.002059915568679571,
+                            0.00219786842353642,
+                            0.002241332782432437,
+                            0.0029454382602125406,
+                            0.0021472384687513113,
+                            0.002435458591207862,
+                            0.002156966831535101,
+                            0.0019780686125159264,
+                            0.0022371329832822084,
+                            0.0019866398070007563,
+                            0.002014638390392065,
+                            0.001987429801374674,
+                            0.0020679838489741087,
+                            0.002068640198558569,
+                            0.0021100782323628664,
+                            0.0019142660312354565,
+                            0.0026210458017885685,
+                            0.0021151448599994183,
+                            0.0023149808403104544,
+                            0.001765275839716196,
+                            0.0028209604788571596,
+                            0.0020816957112401724,
+                            0.0022061667405068874,
+                            0.002135232090950012,
+                            0.0018210868583992124,
+                            0.0023001437075436115,
+                            0.0021450109779834747,
+                            0.0019631334580481052,
+                            0.002153230831027031,
+                            0.0022106696851551533,
+                            0.002267332747578621,
+                            0.002135823480784893,
+                            0.002090109745040536,
+                            0.0021341389510780573,
+                            0.0020355074666440487,
+                            0.0015955889830365777,
+                            0.0020649542566388845,
+                            0.0019545559771358967,
+                            0.001691710203886032,
+                            0.0022275871597230434,
+                            0.0018411381170153618,
+                            0.0021231744904071093,
+                            0.0020730639807879925,
+                            0.0018911130027845502,
+                            0.0020293667912483215,
+                            0.0019049618858844042,
+                            0.002734201727434993,
+                            0.002130514709278941,
+                            0.0018843838479369879,
+                            0.0023450099397450686,
+                            0.002253177110105753,
+                            0.0019138832576572895,
+                            0.0020785320084542036,
+                            0.0021414565853774548,
+                            0.0020741112530231476,
+                            0.002128737047314644,
+                            0.0021227903198450804,
+                            0.0025254678912460804,
+                            0.002127330983057618,
+                            0.0016888466197997332,
+                            0.001609320635907352,
+                            0.001854416448622942,
+                            0.002436218783259392,
+                            0.0018479021964594722,
+                            0.002047653077170253,
+                            0.002089957008138299,
+                            0.0021100721787661314,
+                            0.0020358487963676453,
+                            0.0023592738434672356,
+                            0.0022008398082107306,
+                            0.002224762225523591,
+                            0.002017814666032791,
+                            0.0020703203044831753,
+                            0.00217367778532207,
+                            0.0021283579990267754,
+                            0.0020320280455052853,
+                            0.002115170704200864,
+                            0.002326027024537325,
+                            0.0025247528683394194,
+                            0.0020710863173007965,
+                            0.002070352900773287,
+                            0.0021203302312642336,
+                            0.0021023268345743418,
+                            0.0017012656899169087,
+                            0.0017256407300010324,
+                            0.0019010233227163553,
+                            0.002492871368303895,
+                            0.0018640015041455626,
+                            0.002364959567785263,
+                            0.0020645041950047016,
+                            0.001825492363423109,
+                            0.0028603856917470694,
+                            0.0018470505019649863,
+                            0.0019181163515895605,
+                            0.002215324668213725,
+                            0.002010710770264268,
+                            0.0021376332733780146,
+                            0.002308401744812727,
+                            0.0018087595235556364,
+                            0.0018731328891590238,
+                            0.002169393701478839,
+                            0.002079957164824009,
+                            0.0021668518893420696,
+                            0.0022472329437732697,
+                            0.002190721221268177,
+                            0.002173496875911951,
+                            0.0021086824126541615,
+                            0.001903065713122487,
+                            0.002201864728704095,
+                            0.002146098529919982,
+                            0.002098140073940158,
+                            0.0019101902144029737,
+                            0.0019922545179724693,
+                            0.001915078959427774,
+                            0.0019900219049304724,
+                            0.0023240302689373493,
+                            0.0021959703881293535,
+                            0.0021705152466893196,
+                            0.002140637021511793,
+                            0.002524706069380045,
+                            0.0017572565702721477,
+                            0.002055431716144085,
+                            0.002117004245519638,
+                            0.0020094069186598063,
+                            0.0020638129208236933,
+                            0.0020970692858099937,
+                            0.0025508778635412455,
+                            0.002071948954835534,
+                            0.0018353949999436736,
+                            0.0020874880719929934,
+                            0.002170722233131528,
+                            0.001794992247596383,
+                            0.0017913844203576446,
+                            0.0018132601398974657,
+                            0.002070487942546606,
+                            0.0021415052469819784,
+                            0.002192809944972396,
+                            0.002195883309468627,
+                            0.0020887134596705437,
+                            0.002083614468574524,
+                            0.0019921129569411278,
+                            0.002052236581221223,
+                            0.0020798188634216785,
+                            0.0019047022797167301,
+                            0.002243815455585718
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 1,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.05773673206567764
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "dense_arch.model._mlp.0",
+                "input_tensor_infos": [
+                    {
+                        "id": 1,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.05773673206567764
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 2,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.05722489953041077
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 2,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.05722489953041077
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "dense_arch:model:_mlp:1": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "dense_arch.model._mlp.1._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 2,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.05722489953041077
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0003814205701928586,
+                            0.0003856203402392566,
+                            0.001103446469642222,
+                            0.0008714395225979388,
+                            0.00037817127304151654,
+                            0.00036842000554315746,
+                            0.0004113488830626011,
+                            0.00038186824531294405,
+                            0.0003826507308986038,
+                            0.0012262117816135287,
+                            0.0007383538759313524,
+                            0.0007125691045075655,
+                            0.0003837271360680461,
+                            0.0006967199151404202,
+                            0.00043902298784814775,
+                            0.0014510152395814657,
+                            0.00041987517033703625,
+                            0.0005524512380361557,
+                            0.0003831146168522537,
+                            0.0013337649870663881,
+                            0.0003822623984888196,
+                            0.0004245903983246535,
+                            0.00042021003901027143,
+                            0.000383917213184759,
+                            0.0008418288198299706,
+                            0.000383725855499506,
+                            0.00038499143556691706,
+                            0.0010402490152046084,
+                            0.0012052443344146013,
+                            0.0016367575153708458,
+                            0.0003844212624244392,
+                            0.0008826357079669833,
+                            0.00038253801176324487,
+                            0.0009841268183663487,
+                            0.0012090025702491403,
+                            0.0012213094159960747,
+                            0.0003833434311673045,
+                            0.00038207421312108636,
+                            0.00038708135252818465,
+                            0.0017759704496711493,
+                            0.0011546585010364652,
+                            0.00038241108995862305,
+                            0.0003855624236166477,
+                            0.0003833860391750932,
+                            0.0003853632078971714,
+                            0.00038545357529073954,
+                            0.00042899345862679183,
+                            0.0004226507153362036,
+                            0.0007937821792438626,
+                            0.0003853772359434515,
+                            0.00037010639789514244,
+                            0.00038458133349195123,
+                            0.00038378042518161237,
+                            0.000368787586921826,
+                            0.0013543401146307588,
+                            0.00042292484431527555,
+                            0.001071740873157978,
+                            0.00037853000685572624,
+                            0.0011320807971060276,
+                            0.0010547033743932843,
+                            0.0004051238938700408,
+                            0.0013776407577097416,
+                            0.001105888863094151,
+                            0.001686673378571868,
+                            0.0003796137752942741,
+                            0.00040738159441389143,
+                            0.00038511137245222926,
+                            0.00034700939431786537,
+                            0.00038367643719539046,
+                            0.0010195522336289287,
+                            0.0003957435837946832,
+                            0.001250236527994275,
+                            0.00040693709161132574,
+                            0.0003997977473773062,
+                            0.00136468221899122,
+                            0.001276098657399416,
+                            0.0009081555181182921,
+                            0.0020354469306766987,
+                            0.00037401431472972035,
+                            0.0004346101777628064,
+                            0.0003738317755050957,
+                            0.0003781073319260031,
+                            0.0004314974357839674,
+                            0.0009172310819849372,
+                            0.0003830826608464122,
+                            0.0003801020502578467,
+                            0.0014421362429857254,
+                            0.0004551030870061368,
+                            0.0004529067373368889,
+                            0.0007889969274401665,
+                            0.0003795662778429687,
+                            0.00047587993321940303,
+                            0.0003830471250694245,
+                            0.000424789177486673,
+                            0.0010026234667748213,
+                            0.00038399259210564196,
+                            0.00047910489956848323,
+                            0.0007839479367248714,
+                            0.0014256623107939959,
+                            0.00038243093877099454,
+                            0.000640947197098285,
+                            0.0003787996538449079,
+                            0.00034508566022850573,
+                            0.000434366607805714,
+                            0.001230587949976325,
+                            0.00040226709097623825,
+                            0.0004225957382004708,
+                            0.00038183346623554826,
+                            0.000383175938623026,
+                            0.0004522630770225078,
+                            0.00042068370385095477,
+                            0.00038099492667242885,
+                            0.00038580148248001933,
+                            0.00038289299118332565,
+                            0.00038448648410849273,
+                            0.0004488271661102772,
+                            0.0003822983999270946,
+                            0.00038563075941056013,
+                            0.001194117940030992,
+                            0.0003907018108293414,
+                            0.0003936937137041241,
+                            0.001208227826282382,
+                            0.00037369324127212167,
+                            0.0010909559205174446,
+                            0.000979097792878747,
+                            0.001504589687101543,
+                            0.00038129015592858195,
+                            0.0003805749292951077,
+                            0.0006601195782423019,
+                            0.0008330027339980006,
+                            0.0003820141137111932,
+                            0.0004799688467755914,
+                            0.0003852187655866146,
+                            0.0003828879271168262,
+                            0.0014232553075999022,
+                            0.00037889962550252676,
+                            0.0008778349147178233,
+                            0.00038394168950617313,
+                            0.0003935399872716516,
+                            0.0003800170961767435,
+                            0.0014243166660889983,
+                            0.0003815290401689708,
+                            0.00038560538087040186,
+                            0.0009274158510379493,
+                            0.00038505159318447113,
+                            0.0003809893678408116,
+                            0.0003839894779957831,
+                            0.0013848760863766074,
+                            0.0004346009227447212,
+                            0.00039849101449362934,
+                            0.00038803720963187516,
+                            0.0003761562693398446,
+                            0.001260764547623694,
+                            0.0004545408592093736,
+                            0.0003846747858915478,
+                            0.0003849912900477648,
+                            0.0004252018698025495,
+                            0.0012541341129690409,
+                            0.0003857529954984784,
+                            0.0016409403178840876,
+                            0.0003789163311012089,
+                            0.0016067598480731249,
+                            0.0009141663322225213,
+                            0.000374000781448558,
+                            0.00042348631541244686,
+                            0.0003844456805381924,
+                            0.00034644402330741286,
+                            0.00038519164081662893,
+                            0.0003808142791967839,
+                            0.002011967357248068,
+                            0.001723551657050848,
+                            0.0009261839441023767,
+                            0.0011930224718526006,
+                            0.0009809694020077586,
+                            0.00038565235445275903,
+                            0.0008149099303409457,
+                            0.001417910447344184,
+                            0.0003459857252892107,
+                            0.0003465602349024266,
+                            0.00045294789015315473,
+                            0.00034661797690205276,
+                            0.00046107612433843315,
+                            0.0006530711543746293,
+                            0.00039454762008972466,
+                            0.00041676528053358197,
+                            0.00038231100188568234,
+                            0.0003799613332375884,
+                            0.0004080399521626532,
+                            0.001290371990762651,
+                            0.0003865569888148457,
+                            0.0006661736406385899,
+                            0.00042965891771018505,
+                            0.0012916137930005789,
+                            0.00038034518365748227,
+                            0.0003835330717265606,
+                            0.0003812621580436826,
+                            0.0003780377155635506,
+                            0.00039571692468598485,
+                            0.0003798100515268743,
+                            0.0004252730286680162,
+                            0.0012916725827381015,
+                            0.0010914030717685819,
+                            0.0003830015193670988,
+                            0.0003884201869368553,
+                            0.00043153384467586875,
+                            0.0003929086378775537,
+                            0.00041415137820877135,
+                            0.00039834860945120454,
+                            0.0009709449950605631,
+                            0.0003873156092595309,
+                            0.0003896155976690352,
+                            0.001223319792188704,
+                            0.0012829824117943645,
+                            0.00038143276469781995,
+                            0.0018137117149308324,
+                            0.0004104585968889296,
+                            0.0004052842559758574,
+                            0.0009711675811558962,
+                            0.0009091336978599429,
+                            0.0003991298726759851,
+                            0.00038456046604551375,
+                            0.0011575634125620127,
+                            0.00036757340421900153,
+                            0.00039868673775345087,
+                            0.00038361031329259276,
+                            0.0013099083444103599,
+                            0.0004055474710185081,
+                            0.00038253949605859816,
+                            0.0004190717882011086,
+                            0.0012309050653129816,
+                            0.00042330671567469835,
+                            0.0003843137819785625,
+                            0.0014916859799996018,
+                            0.0009410077473148704,
+                            0.0003697953943628818,
+                            0.00038456509355455637,
+                            0.0003733470803126693,
+                            0.00038286185008473694,
+                            0.0003850008361041546,
+                            0.00048224712372757494,
+                            0.0010179064702242613,
+                            0.00038522566319443285,
+                            0.0013346171472221613,
+                            0.0005472212797030807,
+                            0.0003857304691337049,
+                            0.0008163604652509093,
+                            0.0003791423514485359,
+                            0.0003837695112451911,
+                            0.000383604085072875,
+                            0.0009845435852184892,
+                            0.0008484618738293648,
+                            0.0003857271804008633,
+                            0.00038532461621798575,
+                            0.0005367251578718424,
+                            0.00038479940849356353,
+                            0.0003853695816360414
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 3,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.044601019471883774
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "dense_arch.model._mlp.1",
+                "input_tensor_infos": [
+                    {
+                        "id": 3,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.044601019471883774
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 4,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.04854346439242363
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 4,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.04854346439242363
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "dense_arch:model:_mlp:2": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "dense_arch.model._mlp.2._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 4,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.04854346439242363
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0013910909183323383,
+                            0.0014799239579588175,
+                            0.0009417493129149079,
+                            0.001563972793519497,
+                            0.001051687402650714,
+                            0.0006314919446595013,
+                            0.0018558365991339087,
+                            0.0008566948235966265,
+                            0.0004892059951089323,
+                            0.0032512552570551634,
+                            0.0016270888736471534,
+                            0.0010049514239653945,
+                            0.0005284972721710801,
+                            0.0005221585161052644,
+                            0.0005708932876586914,
+                            0.0005730590783059597,
+                            0.0010962176602333784,
+                            0.0013496605679392815,
+                            0.0021498159039765596,
+                            0.000777900218963623,
+                            0.0007879168260842562,
+                            0.0013430631952360272,
+                            0.0005203924374654889,
+                            0.0006746455328539014,
+                            0.001202031853608787,
+                            0.0005666201468557119,
+                            0.0013673516223207116,
+                            0.0012357296654954553,
+                            0.0007359993178397417,
+                            0.0012674934696406126,
+                            0.0008290227269753814,
+                            0.001214474206790328,
+                            0.001191372866742313,
+                            0.0006476813578046858,
+                            0.0007052742876112461,
+                            0.0006191146676428616,
+                            0.0005246601067483425,
+                            0.0008234476554207504,
+                            0.0013688808539882302,
+                            0.001361734583042562,
+                            0.0022184683475643396,
+                            0.0016009577084332705,
+                            0.001140796230174601,
+                            0.0005182682070881128,
+                            0.0007423642091453075,
+                            0.0013491890858858824,
+                            0.0009760895045474172,
+                            0.0006922174943611026,
+                            0.001845178077928722,
+                            0.0013139002257958055,
+                            0.0009710575104691088,
+                            0.0009355126530863345,
+                            0.00097748264670372,
+                            0.0016726356698200107,
+                            0.0005214369739405811,
+                            0.0007915521855466068,
+                            0.0013413111446425319,
+                            0.0011569567723199725,
+                            0.0008787246770225465,
+                            0.0005258596502244473,
+                            0.000987693783827126,
+                            0.0016287286998704076,
+                            0.0006541382754221559,
+                            0.0009019988356158137,
+                            0.0010836818255484104,
+                            0.0011194046819582582,
+                            0.0006940715829841793,
+                            0.001026054611429572,
+                            0.0017925151623785496,
+                            0.0013241581618785858,
+                            0.0007960334187373519,
+                            0.0014090532204136252,
+                            0.0012547456426545978,
+                            0.0021696961484849453,
+                            0.001520531834103167,
+                            0.0008080829284153879,
+                            0.0008374472381547093,
+                            0.0006078359438106418,
+                            0.0005844144616276026,
+                            0.0005340739153325558,
+                            0.0008933785138651729,
+                            0.0009371751802973449,
+                            0.001059233327396214,
+                            0.0007825875654816628,
+                            0.0008585767936892807,
+                            0.0007533769821748137,
+                            0.0010679183760657907,
+                            0.0017051557078957558,
+                            0.0012647506082430482,
+                            0.0014353594742715359,
+                            0.0007032635039649904,
+                            0.0010318418499082327,
+                            0.0005382395465858281,
+                            0.0005556087708100677,
+                            0.0008773534209467471,
+                            0.0011157991830259562,
+                            0.000980847398750484,
+                            0.0006729523302055895,
+                            0.0011948436731472611,
+                            0.0017532246420159936,
+                            0.0005266261287033558,
+                            0.0012689029099419713,
+                            0.0006932197138667107,
+                            0.0007941905059851706,
+                            0.0005479002720676363,
+                            0.0014347314136102796,
+                            0.0005442474503070116,
+                            0.0005235374555923045,
+                            0.0023783338256180286,
+                            0.00198438111692667,
+                            0.0014055399224162102,
+                            0.0007246094173751771,
+                            0.0019253145437687635,
+                            0.00068176124477759,
+                            0.0015527677023783326,
+                            0.000934386218432337,
+                            0.0006698191864416003,
+                            0.0012414113152772188,
+                            0.0006205763202160597,
+                            0.001414248370565474,
+                            0.0009708464494906366,
+                            0.0007755814003758132,
+                            0.0009399249101988971,
+                            0.0005243881605565548,
+                            0.0006064198678359389,
+                            0.0006532343104481697,
+                            0.0004998490330763161,
+                            0.000537425628863275
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 5,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.008966227062046528
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "dense_arch.model._mlp.2",
+                "input_tensor_infos": [
+                    {
+                        "id": 5,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.008966227062046528
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 6,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.00934610515832901
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 6,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.00934610515832901
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "inter_arch": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.V_linears.0",
+                "input_tensor_infos": [
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.03068052977323532
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0007224899018183351,
+                            0.0007444857037626207,
+                            0.0010785674676299095,
+                            0.0008304960792884231,
+                            0.0007089514983817935,
+                            0.0009914626134559512,
+                            0.0009091422543860972,
+                            0.000679816585034132,
+                            0.0010102398227900267,
+                            0.0009229014394804835,
+                            0.0011965262237936258,
+                            0.0006699389778077602,
+                            0.0007607698789797723,
+                            0.0007521064835600555,
+                            0.0008468159940093756,
+                            0.0007088689599186182,
+                            0.0009686272824183106,
+                            0.0007801342871971428,
+                            0.001298951217904687,
+                            0.000938365759793669,
+                            0.0007879441254772246,
+                            0.0006914357072673738,
+                            0.0008145557949319482,
+                            0.0009570136899128556,
+                            0.0009317446383647621,
+                            0.0009940351592376828,
+                            0.0006934481207281351,
+                            0.0008641786407679319,
+                            0.0007126000127755105,
+                            0.0009714633924886584,
+                            0.0006408952758647501,
+                            0.0011580210411921144,
+                            0.0007923395605757833,
+                            0.0007947187405079603,
+                            0.0007944109966047108,
+                            0.0007573838229291141,
+                            0.000833001162391156,
+                            0.0007776030688546598,
+                            0.0007982041570357978,
+                            0.0007141837268136442,
+                            0.0007728826603852212,
+                            0.0008470327593386173,
+                            0.0009667308768257499,
+                            0.0008341664797626436,
+                            0.000900680897757411,
+                            0.001035906607285142,
+                            0.0008630981901660562,
+                            0.0010462161153554916,
+                            0.0011540265986695886,
+                            0.000738036644179374,
+                            0.0009265323751606047,
+                            0.0006490419036708772,
+                            0.000737060618121177,
+                            0.000791799568105489,
+                            0.0009632249712012708,
+                            0.0006813427899032831,
+                            0.0007367533398792148,
+                            0.0008973156218416989,
+                            0.0006475643604062498,
+                            0.00079604412894696,
+                            0.0008152489899657667,
+                            0.0008932276396080852,
+                            0.0007837412413209677,
+                            0.0006990822148509324,
+                            0.0007251379429362714,
+                            0.0008619811851531267,
+                            0.0009882025187835097,
+                            0.0007666069432161748,
+                            0.0009502913453616202,
+                            0.0006483598845079541,
+                            0.0007609125459566712,
+                            0.0007478449842892587,
+                            0.0010870009427890182,
+                            0.0007203076384030282,
+                            0.0006702210521325469,
+                            0.0007962585659697652,
+                            0.0007582399412058294,
+                            0.00062871805857867,
+                            0.0008289598044939339,
+                            0.0008990783244371414,
+                            0.0007117502973414958,
+                            0.000743058102671057,
+                            0.0008231333340518177,
+                            0.0007038985495455563,
+                            0.0006890983786433935,
+                            0.0007165804272517562,
+                            0.0009532508556731045,
+                            0.0007438199245370924,
+                            0.0008919921820051968,
+                            0.0010083524975925684,
+                            0.0008117058896459639,
+                            0.0008661613683216274,
+                            0.0006471690721809864,
+                            0.0010138413636013865,
+                            0.000915012788027525,
+                            0.000869423383846879,
+                            0.0008763379300944507,
+                            0.001264840830117464,
+                            0.0010045042727142572,
+                            0.0007046740502119064,
+                            0.0009308262378908694,
+                            0.0008777580223977566,
+                            0.0008933487697504461,
+                            0.0007192257908172905,
+                            0.000967477448284626,
+                            0.0007316216360777617,
+                            0.0007831009570509195,
+                            0.00084463827079162,
+                            0.0006809425540268421,
+                            0.001414589467458427,
+                            0.0008689579553902149,
+                            0.0011103216093033552,
+                            0.0009823828004300594,
+                            0.000868219998665154,
+                            0.0008818802889436483,
+                            0.0008022155379876494,
+                            0.0010151306632906199,
+                            0.001658021123148501,
+                            0.000765566888730973,
+                            0.000876921636518091,
+                            0.0008691889815963805,
+                            0.0007645676960237324,
+                            0.000745923665817827,
+                            0.0011651207460090518,
+                            0.0009713663603179157,
+                            0.0006728625157848,
+                            0.0006741394172422588,
+                            0.0008115637465380132,
+                            0.0008042796398513019,
+                            0.0013184044510126114,
+                            0.0009416433167643845,
+                            0.0007552349125035107,
+                            0.0006801084382459521,
+                            0.0010763126192614436,
+                            0.000672965485136956,
+                            0.0007456628372892737,
+                            0.0007091052830219269,
+                            0.0007939295610412955,
+                            0.0010966839035972953,
+                            0.0007494095480069518,
+                            0.0010706775356084108,
+                            0.0009506331407465041,
+                            0.0007130256271921098,
+                            0.0009107174701057374,
+                            0.0008753269794397056,
+                            0.001053221058100462,
+                            0.0007577462820336223,
+                            0.0007010528934188187,
+                            0.0007825610227882862,
+                            0.0013467328390106559,
+                            0.0011145089520141482,
+                            0.0007258603000082076,
+                            0.0007587188738398254,
+                            0.0011272431584075093,
+                            0.0008888538577593863,
+                            0.001211225870065391,
+                            0.0009481008164584637,
+                            0.0009107395308092237,
+                            0.000687237421516329,
+                            0.0008990118512883782,
+                            0.0009450465440750122,
+                            0.0009452897356823087,
+                            0.0007008220418356359,
+                            0.0008629641379229724,
+                            0.0007148963049985468,
+                            0.0008639252046123147,
+                            0.0008515724912285805,
+                            0.0011633080430328846,
+                            0.0009125055512413383,
+                            0.0006474363035522401,
+                            0.0007998345536179841,
+                            0.0007459612679667771,
+                            0.0008184458711184561,
+                            0.0013867357047274709,
+                            0.000770506274420768,
+                            0.0006980307516641915,
+                            0.0007357778958976269,
+                            0.0007476788014173508,
+                            0.0010792972752824426,
+                            0.0007631723419763148,
+                            0.0008488388848491013,
+                            0.0007056772010400891,
+                            0.000911479932256043,
+                            0.0013148859143257141,
+                            0.0007742431480437517,
+                            0.0006923768669366837,
+                            0.0007258416735567153,
+                            0.0007161234389059246,
+                            0.0006970493122935295,
+                            0.0008224659832194448,
+                            0.0009545040666125715,
+                            0.0009034839458763599,
+                            0.0007188183371908963,
+                            0.000855113728903234,
+                            0.000821631052531302,
+                            0.00067783088888973,
+                            0.0006474335677921772,
+                            0.0007864517974667251,
+                            0.0008840923546813428,
+                            0.0006631092401221395,
+                            0.0009415387758053839,
+                            0.0008650662493892014,
+                            0.0008701069746166468,
+                            0.0007092505693435669,
+                            0.001024395925924182,
+                            0.0008786309626884758,
+                            0.0009164402144961059,
+                            0.0006979623576626182,
+                            0.0008157112752087414,
+                            0.0008588808705098927,
+                            0.0008752238936722279,
+                            0.0007269677007570863,
+                            0.0007620592368766665,
+                            0.0007387535297311842,
+                            0.0010133145842701197,
+                            0.0007981800590641797,
+                            0.0008611847297288477,
+                            0.0008142807055264711,
+                            0.0007741790032014251,
+                            0.0010040946071967483,
+                            0.0007265154854394495,
+                            0.0009781057015061378,
+                            0.0008140174322761595,
+                            0.0007419412140734494,
+                            0.0007665016455575824,
+                            0.0011981897987425327,
+                            0.0006777714006602764,
+                            0.0006565506919287145,
+                            0.000899441831279546,
+                            0.001079535111784935,
+                            0.0007268093759194016,
+                            0.000807644915767014,
+                            0.0007088807178661227,
+                            0.0011151161743327975,
+                            0.0006893572281114757,
+                            0.0008026600116863847,
+                            0.0006678328500129282,
+                            0.0006602767971344292,
+                            0.0008593368693254888,
+                            0.000722787226550281,
+                            0.0006234744796529412,
+                            0.0007417546003125608,
+                            0.00068084237864241,
+                            0.0007767042843624949,
+                            0.0008201932650990784,
+                            0.0007101477822288871,
+                            0.000741739000659436,
+                            0.0009321426623500884,
+                            0.0008880985551513731,
+                            0.0008536202949471772,
+                            0.0009482083842158318,
+                            0.0008518281392753124,
+                            0.0007580122910439968,
+                            0.0011245354544371367,
+                            0.0006940995808690786,
+                            0.0011150138452649117,
+                            0.0007592107285745442,
+                            0.0010043669026345015,
+                            0.0011549002956598997,
+                            0.0006577357416972518,
+                            0.0010692673968151212,
+                            0.00088071683421731,
+                            0.0007860640762373805,
+                            0.0007691384525969625,
+                            0.0007646254962310195,
+                            0.0007128884899429977,
+                            0.0006723004044033587,
+                            0.0008459808886982501,
+                            0.0008516588131897151,
+                            0.000711893429979682,
+                            0.0006690458976663649,
+                            0.0007602344849146903,
+                            0.0008094852673821151,
+                            0.0008267541998066008,
+                            0.0007243382278829813,
+                            0.000939221354201436,
+                            0.0006830134079791605,
+                            0.0009220222709700465,
+                            0.0009488246287219226,
+                            0.0007868792163208127,
+                            0.0008032277692109346,
+                            0.0006355036166496575,
+                            0.0010314906248822808,
+                            0.0009148827521130443,
+                            0.0008282435010187328,
+                            0.0008493282948620617,
+                            0.0006117642624303699,
+                            0.0011069962056353688,
+                            0.0008422725950367749,
+                            0.0007930422434583306,
+                            0.0008023917907848954,
+                            0.0009360715048387647,
+                            0.0007273771916516125,
+                            0.0008972944342531264,
+                            0.0010303804883733392,
+                            0.0007799018640071154,
+                            0.0007331584929488599,
+                            0.0008014358463697135,
+                            0.0010494720190763474,
+                            0.0007422984926961362,
+                            0.0007239149999804795,
+                            0.0007571082096546888,
+                            0.0007100031361915171,
+                            0.0008736925083212554,
+                            0.0007656224188394845,
+                            0.000843617192003876,
+                            0.0008578684064559639,
+                            0.0009947341168299317,
+                            0.0010149055160582066,
+                            0.0008115058299154043,
+                            0.000690698332618922,
+                            0.0007856762385927141,
+                            0.0006565725198015571,
+                            0.0010594669729471207,
+                            0.0011040695244446397,
+                            0.0006580721237696707,
+                            0.0007126210257411003,
+                            0.0008028581505641341,
+                            0.0007561222300864756,
+                            0.0009444323368370533,
+                            0.0009322923724539578,
+                            0.0009087790385819972,
+                            0.0007932998123578727,
+                            0.0009087954531423748,
+                            0.0009831310017034411,
+                            0.0007000413024798036,
+                            0.000783652882091701,
+                            0.000693272624630481,
+                            0.0006966539076529443,
+                            0.0008015832281671464,
+                            0.000797309388872236,
+                            0.0007249426562339067,
+                            0.0008759842021390796,
+                            0.0006994326249696314,
+                            0.0008336256141774356,
+                            0.0008115170639939606,
+                            0.0007929040584713221,
+                            0.0007481001666747034,
+                            0.0006691577727906406,
+                            0.0007253999938257039,
+                            0.0009002683800645173,
+                            0.0010895738378167152,
+                            0.001327762845903635,
+                            0.0007966267876327038,
+                            0.0009049728978425264,
+                            0.0006194750894792378,
+                            0.0007169093005359173,
+                            0.0008464023121632636,
+                            0.0007821923936717212,
+                            0.0009487634524703026,
+                            0.0009537508012726903,
+                            0.0008096337551251054,
+                            0.0007050508284009993,
+                            0.0006440657889470458,
+                            0.000726968573872,
+                            0.0008862094837240875,
+                            0.0007637774106115103,
+                            0.0008666471694596112,
+                            0.0011731755221262574,
+                            0.001035245368257165,
+                            0.000896160549018532,
+                            0.0007064257515594363,
+                            0.0014000379014760256,
+                            0.0007923701778054237,
+                            0.001082196249626577,
+                            0.0009497863939031959,
+                            0.0007595287752337754,
+                            0.0008486906881444156,
+                            0.0008479611133225262,
+                            0.000915327575057745,
+                            0.0011086448794230819,
+                            0.0006966002401895821,
+                            0.0010537718189880252,
+                            0.0008423975668847561,
+                            0.0007518080528825521,
+                            0.0010563562391325831,
+                            0.0008608045754954219,
+                            0.0007608923478983343,
+                            0.0009376589441671968,
+                            0.0007672916981391609,
+                            0.000720176612958312,
+                            0.0007674240041524172,
+                            0.0008227373473346233,
+                            0.0008638181025162339,
+                            0.0008879703236743808,
+                            0.0007753929821774364,
+                            0.0007533463649451733,
+                            0.0006531343678943813,
+                            0.0007285717292688787,
+                            0.000986652565188706,
+                            0.000916859891731292,
+                            0.0009280125959776342,
+                            0.0006856642430648208,
+                            0.0009406103054061532,
+                            0.0006610547425225377,
+                            0.0008030686876736581,
+                            0.0007716139662079513,
+                            0.000717483984772116,
+                            0.0006834250525571406,
+                            0.000689470034558326,
+                            0.0008182545425370336,
+                            0.0007855486473999918,
+                            0.0008870811434462667,
+                            0.0008525261073373258,
+                            0.000676182156894356,
+                            0.0006531963008455932,
+                            0.0009881815640255809,
+                            0.0009010365465655923,
+                            0.0006870065699331462,
+                            0.0006943610496819019,
+                            0.0010064623784273863,
+                            0.0011886818101629615,
+                            0.0006784281576983631,
+                            0.0008854951593093574,
+                            0.0006957851001061499,
+                            0.0011700551258400083,
+                            0.0007947076228447258,
+                            0.0006843109149485826,
+                            0.0008321880595758557,
+                            0.0007171452161855996,
+                            0.001278409850783646,
+                            0.000737659283913672,
+                            0.0007722286391071975,
+                            0.0007945048855617642,
+                            0.0008890273165889084,
+                            0.000675543094985187,
+                            0.0007478733896277845,
+                            0.0008845759439282119,
+                            0.0008176531409844756,
+                            0.0006309749442152679,
+                            0.000863811990711838,
+                            0.0010985983535647392,
+                            0.0008469533058814704,
+                            0.0009073474793694913,
+                            0.0006847930490039289,
+                            0.0012079840525984764,
+                            0.0009676638292148709,
+                            0.0007484054076485336,
+                            0.0007687517208978534,
+                            0.0007638945244252682,
+                            0.0006867680349387228,
+                            0.0009600252378731966,
+                            0.00128581200260669,
+                            0.0008114702650345862,
+                            0.000936672615353018,
+                            0.0006954053533263505,
+                            0.0009854899253696203,
+                            0.0009295844356529415,
+                            0.0008853028994053602,
+                            0.0008290970581583679,
+                            0.0009346535080112517,
+                            0.0007446966483257711,
+                            0.0009380452684126794,
+                            0.0007778956787660718,
+                            0.0008706712396815419,
+                            0.0007923712255433202,
+                            0.0007342190947383642,
+                            0.0008453316404484212,
+                            0.0009319140226580203,
+                            0.0011744503863155842,
+                            0.001160113955847919,
+                            0.0007705035386607051,
+                            0.0012248690472915769,
+                            0.0007799264276400208,
+                            0.0008664479828439653,
+                            0.0008251015096902847,
+                            0.0007971930899657309,
+                            0.0008329591364599764,
+                            0.0007880173507146537,
+                            0.0006791359628550708,
+                            0.0009232227457687259,
+                            0.000732466287445277,
+                            0.0009863263694569468,
+                            0.000782198563683778,
+                            0.0007159696542657912,
+                            0.0008625781629234552,
+                            0.0009407996549271047,
+                            0.0009303490514867008,
+                            0.000968555745203048,
+                            0.0007739986758679152,
+                            0.0007620008545927703,
+                            0.0007094002794474363,
+                            0.0007002067868597806,
+                            0.0007900614873506129,
+                            0.0007021049968898296,
+                            0.000768079305998981,
+                            0.0009614984737709165,
+                            0.0010907724499702454,
+                            0.0007605345454066992,
+                            0.0007483722874894738,
+                            0.0011604761239141226,
+                            0.0006606425740756094,
+                            0.0007789032533764839,
+                            0.000928979367017746,
+                            0.0010963944951072335,
+                            0.0008457404910586774,
+                            0.0005955906235612929,
+                            0.000760515860747546,
+                            0.0007393134292215109,
+                            0.0006985858781263232,
+                            0.0008078202954493463,
+                            0.000966880121268332,
+                            0.0010670478222891688,
+                            0.0006972895353101194,
+                            0.001135992701165378,
+                            0.0008008047007024288,
+                            0.0007722931914031506,
+                            0.0007251732749864459,
+                            0.0007003400824032724,
+                            0.0008461668039672077,
+                            0.0006770758773200214,
+                            0.000918074045330286
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 60,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.023884600028395653
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.W_linears.0",
+                "input_tensor_infos": [
+                    {
+                        "id": 60,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.023884600028395653
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0009473703685216606,
+                            0.0010398119920864701,
+                            0.0006112687406130135,
+                            0.0010293483501300216,
+                            0.0010587507858872414,
+                            0.0005366202094592154,
+                            0.0008412479655817151,
+                            0.0005793982418254018,
+                            0.0006494323606602848,
+                            0.0011070563923567533,
+                            0.00069661985617131,
+                            0.000818211876321584,
+                            0.0004708287597168237,
+                            0.0005365453544072807,
+                            0.0005271396366879344,
+                            0.0005168476491235197,
+                            0.000681709498167038,
+                            0.0007605348946526647,
+                            0.0007779764127917588,
+                            0.0005260807811282575,
+                            0.0007050520507618785,
+                            0.0005670820246450603,
+                            0.0006807268364354968,
+                            0.0004890361451543868,
+                            0.0005958633264526725,
+                            0.0006255225162021816,
+                            0.0007633554632775486,
+                            0.0006424227613024414,
+                            0.0005630793748423457,
+                            0.0005606594495475292,
+                            0.0006565130315721035,
+                            0.000801680376753211,
+                            0.0005457429215312004,
+                            0.0006511164829134941,
+                            0.0005806066328659654,
+                            0.0005687277880497277,
+                            0.0006057075224816799,
+                            0.0005450330791063607,
+                            0.0005979748093523085,
+                            0.0005928878090344369,
+                            0.0010072836885228753,
+                            0.0008466170402243733,
+                            0.0008224968914873898,
+                            0.0005198857979848981,
+                            0.0006896228878758848,
+                            0.002002137014642358,
+                            0.0006284744595177472,
+                            0.0006062897155061364,
+                            0.0008162048761732876,
+                            0.0012385182781144977,
+                            0.000704769161529839,
+                            0.0005840279045514762,
+                            0.0006277411594055593,
+                            0.001053006388247013,
+                            0.0006033065728843212,
+                            0.0005236422293819487,
+                            0.0005641926545649767,
+                            0.0006193692679516971,
+                            0.0010048635303974152,
+                            0.0005934652872383595,
+                            0.0006491814856417477,
+                            0.0010872907005250454,
+                            0.0006254927720874548,
+                            0.0007158831576816738,
+                            0.0006582205533050001,
+                            0.0005109861958771944,
+                            0.0005619572475552559,
+                            0.0011916747316718102,
+                            0.0006473463727161288,
+                            0.0007115997141227126,
+                            0.0005466614966280758,
+                            0.0006192517466843128,
+                            0.001027593039907515,
+                            0.0005870994064025581,
+                            0.0008863521507009864,
+                            0.0005819863290525973,
+                            0.0005758680053986609,
+                            0.000650151981972158,
+                            0.0006622139480896294,
+                            0.0005785365356132388,
+                            0.0006491431850008667,
+                            0.0005714329890906811,
+                            0.0006194059969857335,
+                            0.0006719413795508444,
+                            0.0005783809465356171,
+                            0.0006781583651900291,
+                            0.0005737296887673438,
+                            0.000828288437332958,
+                            0.001148446463048458,
+                            0.0005930316401645541,
+                            0.0006071607931517065,
+                            0.0006069413502700627,
+                            0.0005727425450459123,
+                            0.0006476231501437724,
+                            0.00121128442697227,
+                            0.0006315252976492047,
+                            0.0009073440451174974,
+                            0.0005122224101796746,
+                            0.0005239455495029688,
+                            0.0012209187261760235,
+                            0.0005369731807149947,
+                            0.0009098772425204515,
+                            0.0005189631483517587,
+                            0.0007494312012568116,
+                            0.0007160158711485565,
+                            0.00116322364192456,
+                            0.0005481951520778239,
+                            0.0006216820911504328,
+                            0.000909657683223486,
+                            0.0006320906686596572,
+                            0.0006346309673972428,
+                            0.0005795666365884244,
+                            0.0008607981144450605,
+                            0.0006434035021811724,
+                            0.0005718345637433231,
+                            0.0006050835363566875,
+                            0.0005600065924227238,
+                            0.0009816818637773395,
+                            0.000613862241152674,
+                            0.0011448778677731752,
+                            0.0005528996116481721,
+                            0.0007590535096824169,
+                            0.0005816270713694394,
+                            0.0005727295065298676,
+                            0.0008179487776942551,
+                            0.000617257144767791,
+                            0.0006378855905495584,
+                            0.0005574822425842285,
+                            0.0007857977761887014,
+                            0.000609825539868325,
+                            0.0006862960872240365,
+                            0.0008115330128930509,
+                            0.0005454886704683304,
+                            0.0009135157451964915,
+                            0.0005724925431422889,
+                            0.0008199920412153006,
+                            0.0006702419486828148,
+                            0.0006440485594794154,
+                            0.0007113829487934709,
+                            0.0005823586834594607,
+                            0.0005721604684367776,
+                            0.0006895744008943439,
+                            0.0006930278614163399,
+                            0.0005869589513167739,
+                            0.0005242566112428904,
+                            0.0005431240424513817,
+                            0.0006614436279051006,
+                            0.0006285464623942971,
+                            0.0005839485675096512,
+                            0.0007139795343391597,
+                            0.0008075071964412928,
+                            0.0006423882441595197,
+                            0.0006707739667035639,
+                            0.0008977372199296951,
+                            0.0007331888773478568,
+                            0.0005059934919700027,
+                            0.000515132094733417,
+                            0.0005504137370735407,
+                            0.0006023175083100796,
+                            0.0005862178513780236,
+                            0.0006380920531228185,
+                            0.0005795149481855333,
+                            0.0005506526213139296,
+                            0.0005631254753097892,
+                            0.000801401911303401,
+                            0.0006032586097717285,
+                            0.0006157816387712955,
+                            0.0006494536646641791,
+                            0.0006458748830482364,
+                            0.0005548947374336421,
+                            0.0006908352952450514,
+                            0.0006034342222847044,
+                            0.0006742541445419192,
+                            0.0005985815078020096,
+                            0.000545877730473876,
+                            0.0005247752997092903,
+                            0.0005698977620340884,
+                            0.0006209210259839892,
+                            0.0007105597760528326,
+                            0.0007888831896707416,
+                            0.0005080324481241405,
+                            0.0005495644873008132,
+                            0.0006426109466701746,
+                            0.0007593839545734227,
+                            0.0006962153711356223,
+                            0.0005998730775900185,
+                            0.0005969788180664182,
+                            0.0006727323052473366,
+                            0.0006171762943267822,
+                            0.0006347968010231853,
+                            0.0005828204448334873,
+                            0.000724687532056123,
+                            0.000779527414124459,
+                            0.0006228176644071937,
+                            0.0006728806765750051,
+                            0.0005786133697256446,
+                            0.000629601941909641,
+                            0.000633045332506299,
+                            0.0005656991270370781,
+                            0.0006979643949307501,
+                            0.000644027313683182,
+                            0.0006413717055693269,
+                            0.0006507501821033657,
+                            0.0007585118291899562,
+                            0.0006302896654233336,
+                            0.0006694338517263532,
+                            0.0006121443002484739,
+                            0.0005559332785196602,
+                            0.0010055577149614692,
+                            0.0006834650994278491,
+                            0.0006319587118923664,
+                            0.0007337504648603499,
+                            0.0008155849063768983,
+                            0.0006136208539828658,
+                            0.0005658412701450288,
+                            0.0006189591949805617,
+                            0.000617943296674639,
+                            0.0007258998812176287,
+                            0.0006509695085696876,
+                            0.0005146279581822455,
+                            0.0007170502794906497,
+                            0.0007663647993467748,
+                            0.0006321476539596915,
+                            0.0005480016116052866,
+                            0.0006407705950550735,
+                            0.0006411343347281218,
+                            0.0005857408395968378,
+                            0.0007333155372180045,
+                            0.0005417069769464433,
+                            0.0005943233263678849,
+                            0.0007005504448898137,
+                            0.0005642972537316382,
+                            0.0006138525204733014,
+                            0.000602995976805687,
+                            0.0005907198647037148,
+                            0.0006066401838324964,
+                            0.0006363493157550693,
+                            0.0006449033389799297,
+                            0.0006969643873162568,
+                            0.0007559540681540966,
+                            0.0008350660791620612,
+                            0.0006226249388419092,
+                            0.0008049191674217582,
+                            0.0005334440502338111,
+                            0.0007623815326951444,
+                            0.0006717166397720575,
+                            0.000618096673861146,
+                            0.000542478752322495,
+                            0.0005569818895310163,
+                            0.0008790257852524519,
+                            0.00059967499691993,
+                            0.0007251645438373089,
+                            0.0004911334835924208,
+                            0.0007510174764320254,
+                            0.0005600409931503236,
+                            0.0007235254743136466,
+                            0.0007860702462494373,
+                            0.0005247156950645149,
+                            0.0005578129203058779,
+                            0.0005258030141703784,
+                            0.0007524622487835586,
+                            0.0008894862257875502,
+                            0.0006426092004403472,
+                            0.0007011886918917298,
+                            0.0005801310180686414,
+                            0.0005989923956803977,
+                            0.0005591862718574703,
+                            0.0012867660261690617,
+                            0.0007663024589419365,
+                            0.0008890306926332414,
+                            0.000981785706244409,
+                            0.0006164090591482818,
+                            0.0010681911371648312,
+                            0.0006615921738557518,
+                            0.0006662046071141958,
+                            0.0005543229635804892,
+                            0.0011131782084703445,
+                            0.0006284339469857514,
+                            0.0007420041947625577,
+                            0.0007509459974244237,
+                            0.0014949345495551825,
+                            0.0008706131484359503,
+                            0.0014784575905650854,
+                            0.000648070068564266,
+                            0.0006477735587395728,
+                            0.0008673913544043899,
+                            0.0007111128652468324,
+                            0.0006582645582966506,
+                            0.0006022838060744107,
+                            0.0005757013568654656,
+                            0.0007691030623391271,
+                            0.0010230706539005041,
+                            0.0006896955310367048,
+                            0.000591533666010946,
+                            0.000760573660954833,
+                            0.0007279348792508245,
+                            0.0006943723419681191,
+                            0.0010888517135754228,
+                            0.0005426201969385147,
+                            0.0013310443609952927,
+                            0.0008024217677302659,
+                            0.0005633006221614778,
+                            0.0007245841552503407,
+                            0.0006236846675164998,
+                            0.000795442727394402,
+                            0.0006211879081092775,
+                            0.0007297561387531459,
+                            0.0005710700643248856,
+                            0.0007236436940729618,
+                            0.0006931961979717016,
+                            0.0008888354641385376,
+                            0.0005138638080097735,
+                            0.0006668335990980268,
+                            0.0010974227916449308,
+                            0.0009557107696309686,
+                            0.0008959997212514281,
+                            0.0007950513390824199,
+                            0.0006990499095991254,
+                            0.0008773591252975166,
+                            0.0005563699523918331,
+                            0.0006630551069974899,
+                            0.0007889766129665077,
+                            0.0006406063912436366,
+                            0.0005984673043712974,
+                            0.0010408149100840092,
+                            0.0006085666245780885,
+                            0.0005301204510033131,
+                            0.0007426755037158728,
+                            0.0009114235872402787,
+                            0.0007789963274262846,
+                            0.0006622672080993652,
+                            0.0005836617201566696,
+                            0.0006028501084074378,
+                            0.0008562500006519258,
+                            0.0011154605308547616,
+                            0.0011758924229070544,
+                            0.0007898747571744025,
+                            0.0008606105693615973,
+                            0.0014356530737131834,
+                            0.0006777578382752836,
+                            0.0006688638823106885,
+                            0.0007849226822145283,
+                            0.000836559513118118,
+                            0.0008684970671311021,
+                            0.0007132997852750123,
+                            0.00075360486516729,
+                            0.0007855233852751553,
+                            0.0006155117298476398,
+                            0.0005592706729657948,
+                            0.0005517912795767188,
+                            0.0005690395482815802,
+                            0.0007887415122240782,
+                            0.0007965982076711953,
+                            0.0007255828240886331,
+                            0.0005745674134232104,
+                            0.0007398810121230781,
+                            0.0008223692420870066,
+                            0.0006893376121297479,
+                            0.0005657675792463124,
+                            0.0006584328948520124,
+                            0.0009099075687117875,
+                            0.0007378638256341219,
+                            0.0006012618541717529,
+                            0.0005462593398988247,
+                            0.001648197416216135,
+                            0.0007129166042432189,
+                            0.00057421182282269,
+                            0.0006520027527585626,
+                            0.0008113882504403591,
+                            0.0010265740565955639,
+                            0.0007860285695642233,
+                            0.0005886529688723385,
+                            0.0005582953453995287,
+                            0.0005252195987850428,
+                            0.0009247950511053205,
+                            0.000668144493829459,
+                            0.0009888479253277183,
+                            0.0006544693605974317,
+                            0.0005996894906274974,
+                            0.000628194713499397,
+                            0.000733921944629401,
+                            0.0007187938317656517,
+                            0.0007705477182753384,
+                            0.0005528536275960505,
+                            0.001212761620990932,
+                            0.0011650965316221118,
+                            0.0006829149788245559,
+                            0.001009920728392899,
+                            0.0006028912612237036,
+                            0.0005999626009725034,
+                            0.000589389877859503,
+                            0.000691481283865869,
+                            0.0009862403385341167,
+                            0.0006138069438748062,
+                            0.0008485065773129463,
+                            0.0005082069546915591,
+                            0.0010761208832263947,
+                            0.0006565453950315714,
+                            0.0008696717559359968,
+                            0.0011851447634398937,
+                            0.0010399242164567113,
+                            0.0005930537590757012,
+                            0.0008392716408707201,
+                            0.0006600822089239955,
+                            0.0006654362077824771,
+                            0.0008973279036581516,
+                            0.001267606159672141,
+                            0.0006745342398062348,
+                            0.0008118373225443065,
+                            0.0011913293274119496,
+                            0.001066234428435564,
+                            0.0006910759257152677,
+                            0.0008397631463594735,
+                            0.0007669698679819703,
+                            0.0009328789310529828,
+                            0.0005929634207859635,
+                            0.0016016566660255194,
+                            0.0009127212106250226,
+                            0.0008769460837356746,
+                            0.0010827980004251003,
+                            0.0006544239004142582,
+                            0.0007923838566057384,
+                            0.0011524076107889414,
+                            0.0012161140330135822,
+                            0.0006320991669781506,
+                            0.0009605032973922789,
+                            0.0008492375491186976,
+                            0.0006845417665317655,
+                            0.0006572067504748702,
+                            0.0010865468066185713,
+                            0.0007979451329447329,
+                            0.0015615496085956693,
+                            0.000878794293384999,
+                            0.0006845591706223786,
+                            0.000763369956985116,
+                            0.0013245271984487772,
+                            0.0006843248847872019,
+                            0.0011082279961556196,
+                            0.000792149396147579,
+                            0.0007067342521622777,
+                            0.0007640467374585569,
+                            0.0006864371825940907,
+                            0.0007069516577757895,
+                            0.0007750428048893809,
+                            0.0008060622494667768,
+                            0.0008195432601496577,
+                            0.0007828816887922585,
+                            0.000756298890337348,
+                            0.0007023010984994471,
+                            0.0005488081369549036,
+                            0.000644329353235662,
+                            0.0008314372971653938,
+                            0.0010137202916666865,
+                            0.0008989715715870261,
+                            0.0006966522778384387,
+                            0.0006788625614717603,
+                            0.0007270274800248444,
+                            0.000568706018384546,
+                            0.0009098991868086159,
+                            0.001046928926371038,
+                            0.0018914116080850363,
+                            0.0006263479008339345,
+                            0.0009044701000675559,
+                            0.0010886962991207838,
+                            0.0007031264249235392,
+                            0.0007007158710621297,
+                            0.0008838447974994779,
+                            0.0011084075085818768,
+                            0.0008316808380186558,
+                            0.0007011318812146783,
+                            0.0011667755898088217,
+                            0.0008729705004952848,
+                            0.0009011757210828364,
+                            0.0008819396607577801,
+                            0.0005856735515408218,
+                            0.0010289463680237532,
+                            0.0007929381099529564,
+                            0.0005610556108877063,
+                            0.0008260905160568655,
+                            0.001049663289450109,
+                            0.0007434996077790856,
+                            0.0006273366743698716,
+                            0.0007978228968568146,
+                            0.0012209434062242508,
+                            0.0005910168401896954,
+                            0.0005946559831500053,
+                            0.0005150883225724101,
+                            0.0007828844827599823,
+                            0.0006740267272107303,
+                            0.0006821314455009997,
+                            0.0006174591835588217,
+                            0.0006369919865392148,
+                            0.0005689995596185327,
+                            0.0005887767183594406,
+                            0.0006005127797834575,
+                            0.001345809199847281,
+                            0.0005219730664975941,
+                            0.0006179847987368703,
+                            0.0012447144836187363,
+                            0.0006782123236916959,
+                            0.0014778942568227649,
+                            0.0006501037278212607,
+                            0.0007965286495164037,
+                            0.000699692580383271,
+                            0.0009044545004144311,
+                            0.0011048874584957957,
+                            0.0008299034088850021,
+                            0.0006130648544058204,
+                            0.0007984510739333928,
+                            0.0009021838195621967,
+                            0.0007141805253922939,
+                            0.0012859798735007644,
+                            0.0005634911358356476,
+                            0.0006609369884245098,
+                            0.0005918035167269409,
+                            0.0006198639748618007,
+                            0.0005607097991742194,
+                            0.0005394823965616524,
+                            0.0006293852347880602,
+                            0.0005601362208835781,
+                            0.0005748470430262387,
+                            0.000724041776265949,
+                            0.0005873580230399966,
+                            0.0006411937065422535,
+                            0.000545989372767508,
+                            0.0006322177359834313,
+                            0.0005393329192884266,
+                            0.0005855302442796528,
+                            0.0005587817286141217,
+                            0.0005858521908521652,
+                            0.0006923904293216765,
+                            0.0005631415988318622,
+                            0.0005093096406199038,
+                            0.000604156288318336,
+                            0.0006134569412097335,
+                            0.0005657295696437359,
+                            0.0005662264302372932,
+                            0.0005783814121969044,
+                            0.0006544924108311534,
+                            0.0006853859522379935,
+                            0.0006418878911063075,
+                            0.0006202468648552895,
+                            0.0005827276618219912,
+                            0.0006301066605374217,
+                            0.0005679550813511014,
+                            0.0006192636792548001,
+                            0.0005257338052615523,
+                            0.0005689882091246545,
+                            0.0005993618397042155,
+                            0.0004970147856511176,
+                            0.0006671994924545288,
+                            0.0006623475346714258,
+                            0.0006045556510798633,
+                            0.0006522021722048521,
+                            0.0006170017877593637,
+                            0.0007122954120859504,
+                            0.00045640714233741164,
+                            0.0007109797443263233,
+                            0.0005043493001721799,
+                            0.0005845466512255371,
+                            0.0006923131877556443,
+                            0.0005380800575949252,
+                            0.0005204358603805304,
+                            0.0006054118857719004,
+                            0.0006164266378618777,
+                            0.0005772074218839407,
+                            0.0005528007750399411,
+                            0.0005823816754855216,
+                            0.0006600656197406352,
+                            0.000635602162219584,
+                            0.00057245820062235,
+                            0.0006371531635522842,
+                            0.0005389759899117053,
+                            0.0005250233807601035,
+                            0.0005463458364829421,
+                            0.0005380898364819586,
+                            0.000573357567191124,
+                            0.0007564402767457068,
+                            0.0005536348908208311,
+                            0.0006191596621647477,
+                            0.0004953169263899326,
+                            0.0007122790557332337,
+                            0.0005378709174692631,
+                            0.0007856980664655566,
+                            0.0007764001493342221,
+                            0.0006251384620554745,
+                            0.0005683577619493008,
+                            0.0005422736867330968,
+                            0.0005921950796619058,
+                            0.0007609202875755727,
+                            0.0007193871424533427,
+                            0.0005463464185595512,
+                            0.0005755743477493525,
+                            0.0005473890923894942,
+                            0.0005457772058434784,
+                            0.0005353197921067476,
+                            0.000548378739040345,
+                            0.0007101042428985238,
+                            0.0005423448164947331,
+                            0.0006080742459744215,
+                            0.0006415242096409202,
+                            0.0005770539282821119,
+                            0.0006300208624452353,
+                            0.0006730330060236156,
+                            0.0005644389311783016,
+                            0.0007577980286441743,
+                            0.0006618396146222949,
+                            0.0005099311820231378,
+                            0.0005037211813032627,
+                            0.0006058503640815616,
+                            0.0005449926247820258,
+                            0.0005741032073274255,
+                            0.0006058376166038215,
+                            0.0005727161769755185,
+                            0.0005839635268785059,
+                            0.0005802441155537963,
+                            0.0005261352634988725,
+                            0.0005617695860564709,
+                            0.000542276247870177,
+                            0.000589897739700973,
+                            0.0006320567918010056,
+                            0.0006056074635125697,
+                            0.0005796357290819287,
+                            0.0005049257888458669,
+                            0.0006510452949441969,
+                            0.0006941405590623617,
+                            0.0006911950767971575,
+                            0.0006333343917503953,
+                            0.0005626817001029849,
+                            0.0004819908062927425,
+                            0.0005899237003177404,
+                            0.0006349154864437878,
+                            0.0005469837924465537,
+                            0.0006766000878997147,
+                            0.0005395330372266471,
+                            0.0005531815113499761,
+                            0.0006634846795350313,
+                            0.0005353353335522115,
+                            0.0005750590353272855,
+                            0.0005919767427258193,
+                            0.0005011161556467414,
+                            0.0006279407534748316,
+                            0.0005552815273404121,
+                            0.0006451777881011367,
+                            0.0007011446286924183,
+                            0.000610211631283164,
+                            0.0005197361460886896,
+                            0.0005640419549308717,
+                            0.000564798538107425,
+                            0.0006609900738112628,
+                            0.0005078719696030021,
+                            0.0005849600420333445,
+                            0.0005768159171566367,
+                            0.0006665725959464908,
+                            0.0005162794841453433,
+                            0.0006368954200297594,
+                            0.000547747069504112,
+                            0.0005367412813939154,
+                            0.0005845871637575328,
+                            0.0005529838963411748,
+                            0.0006085422355681658,
+                            0.0006175503949634731,
+                            0.0005338508053682745,
+                            0.0006165555096231401,
+                            0.000556013488676399,
+                            0.0005097686080262065,
+                            0.0006020395667292178,
+                            0.000515025167260319,
+                            0.0008090438786894083,
+                            0.0005203501787036657,
+                            0.0005554090021178126,
+                            0.0005169347277842462,
+                            0.0005214326083660126,
+                            0.0005714894505217671,
+                            0.0004876735038124025,
+                            0.0005540564307011664,
+                            0.0005271025002002716,
+                            0.000488418503664434,
+                            0.0005306392558850348,
+                            0.0007329866057261825,
+                            0.0005495222867466509,
+                            0.0006532807601615787,
+                            0.0005684706848114729,
+                            0.0006440123543143272,
+                            0.0005976929096505046,
+                            0.0006874550017528236,
+                            0.0005628849030472338,
+                            0.000521712121553719,
+                            0.0005514060612767935,
+                            0.0006268239230848849,
+                            0.000580637133680284,
+                            0.000603515247348696,
+                            0.0006673187599517405,
+                            0.0005023601115681231,
+                            0.0006580547778867185,
+                            0.0006053551915101707,
+                            0.0005520704435184598,
+                            0.0005099760019220412,
+                            0.0005082431598566473,
+                            0.0006536010187119246,
+                            0.0006285235867835581,
+                            0.000576604506932199,
+                            0.0005570734501816332,
+                            0.0006883768364787102,
+                            0.0006962206680327654,
+                            0.0007011464913375676,
+                            0.0005945767625235021,
+                            0.0005634025437757373,
+                            0.000605326087679714,
+                            0.0006252772291190922,
+                            0.0006414268282242119,
+                            0.0006238488713279366,
+                            0.0006511328974738717,
+                            0.0005097815301269293,
+                            0.0005925817531533539,
+                            0.0004609350871760398,
+                            0.0005908595630899072,
+                            0.0005759033956564963,
+                            0.0005565488827414811,
+                            0.0005736121674999595,
+                            0.0005495531368069351,
+                            0.0005244119674898684,
+                            0.0006163960206322372,
+                            0.000500804977491498,
+                            0.0006055590929463506,
+                            0.0006624930538237095,
+                            0.0006599402986466885,
+                            0.0005721939378418028,
+                            0.0005474576028063893,
+                            0.0006745312712155282,
+                            0.0006288534495979548,
+                            0.0005142393638379872,
+                            0.0005442472756840289,
+                            0.0006704054540023208,
+                            0.0005336076719686389,
+                            0.0005557431140914559,
+                            0.0005536107928492129,
+                            0.0005950395134277642,
+                            0.0006104845670051873,
+                            0.0005950723425485194,
+                            0.0006042990717105567,
+                            0.0006330535397864878,
+                            0.000618135672993958,
+                            0.000665780738927424,
+                            0.0006116903969086707,
+                            0.0007114315521903336,
+                            0.0005863993428647518,
+                            0.0006004811148159206,
+                            0.0005064471624791622,
+                            0.0005840210360474885,
+                            0.00048083497676998377,
+                            0.0005605531623587012,
+                            0.0007748195203021169,
+                            0.0006101085455156863,
+                            0.0005878114025108516,
+                            0.0005863417754881084,
+                            0.0005523237632587552,
+                            0.0006024757167324424,
+                            0.0005110721103847027,
+                            0.000519033579621464,
+                            0.0005079642869532108,
+                            0.0005635268171317875,
+                            0.0005830656737089157,
+                            0.0005124991293996572,
+                            0.0005853023030795157,
+                            0.0004674936062656343,
+                            0.0006180491182021797,
+                            0.0005224425694905221,
+                            0.0006241851951926947,
+                            0.0005941299023106694,
+                            0.00048521559801883996,
+                            0.0005767341936007142,
+                            0.0005270548281259835,
+                            0.0005824353429488838,
+                            0.0005946989986114204,
+                            0.0005806866101920605,
+                            0.0005962165887467563,
+                            0.0008115133387036622,
+                            0.0005732265417464077,
+                            0.0006535296561196446,
+                            0.0006247539422474802,
+                            0.0005831738817505538,
+                            0.0006916779093444347,
+                            0.0007881455239839852,
+                            0.0004945016698911786,
+                            0.0006196682224981487,
+                            0.0006343668210320175,
+                            0.0007015432929620147,
+                            0.0005126571049913764,
+                            0.0005492026102729142,
+                            0.0007200621184892952,
+                            0.0006545342039316893,
+                            0.0004981577512808144,
+                            0.0005994471721351147,
+                            0.0006626684335060418,
+                            0.0005612304084934294,
+                            0.0006968026864342391,
+                            0.0006393652874976397,
+                            0.0006911114905960858,
+                            0.0005185068002901971,
+                            0.0008232218096964061,
+                            0.0005296944873407483,
+                            0.0005093705840408802,
+                            0.0005614251713268459,
+                            0.0004822084156330675,
+                            0.0005258885794319212,
+                            0.0007014880538918078,
+                            0.0005670136888511479,
+                            0.0005966668832115829,
+                            0.0007065913523547351,
+                            0.000471383158583194,
+                            0.0005352615844458342,
+                            0.0006364126456901431,
+                            0.0005491292686201632,
+                            0.0006119892350398004,
+                            0.0004845980438403785,
+                            0.0006117005250416696,
+                            0.0006864782772026956,
+                            0.0007798276492394507,
+                            0.0006615188322030008,
+                            0.0006445673061534762,
+                            0.0006562079070135951,
+                            0.0005160035216249526,
+                            0.0006098796729929745,
+                            0.0006869424832984805,
+                            0.0006480258889496326,
+                            0.0005139874410815537,
+                            0.0005709226243197918,
+                            0.0005809548310935497,
+                            0.0005858807708136737,
+                            0.0006176820606924593,
+                            0.0005092746578156948,
+                            0.0005976181128062308,
+                            0.0006657822523266077,
+                            0.0006444582832045853,
+                            0.0004858894972130656,
+                            0.00045126170152798295,
+                            0.0006157187162898481,
+                            0.0006178562762215734,
+                            0.0005999113200232387,
+                            0.0006745983264409006,
+                            0.0005469796014949679,
+                            0.0006293128244578838,
+                            0.0007088874117471278,
+                            0.0006031678640283644,
+                            0.0006088726804591715,
+                            0.0005050764884799719,
+                            0.0006128636305220425,
+                            0.0006218887865543365,
+                            0.0006581319612450898,
+                            0.0007290751673281193,
+                            0.0006113633280619979,
+                            0.0005840242956764996,
+                            0.0005350996507331729,
+                            0.0006448247586376965,
+                            0.0005368278361856937,
+                            0.0006528333760797977,
+                            0.0007163583650253713,
+                            0.0005966095486655831,
+                            0.0006003422313369811,
+                            0.0007055274327285588,
+                            0.0005463330890052021,
+                            0.0005901748081669211,
+                            0.0006208120612427592,
+                            0.0006653069285675883,
+                            0.000601593405008316,
+                            0.0006178539479151368,
+                            0.0005760752246715128,
+                            0.000551802571862936,
+                            0.0005657877190969884,
+                            0.000585222733207047,
+                            0.000643955369014293,
+                            0.0005739275366067886,
+                            0.0005517693352885544,
+                            0.0005809040158055723,
+                            0.0005510151386260986,
+                            0.000625076238065958,
+                            0.0005698493914678693,
+                            0.0006308295414783061,
+                            0.0005916978698223829,
+                            0.0007387168589048088,
+                            0.0006001729052513838,
+                            0.0004946293192915618,
+                            0.0005901768454350531,
+                            0.0006375763914547861,
+                            0.0005721884081140161,
+                            0.0005864962586201727,
+                            0.0005952587234787643,
+                            0.0005556449759751558,
+                            0.000535820669028908,
+                            0.0005450399476103485,
+                            0.000678031996358186,
+                            0.0005952556966803968,
+                            0.0006613987497985363,
+                            0.0007191720651462674,
+                            0.0005147968768142164,
+                            0.0008523558499291539,
+                            0.0006998996250331402,
+                            0.0008557819528505206,
+                            0.000721754040569067,
+                            0.000586648122407496,
+                            0.0008759757620282471,
+                            0.0005702702328562737,
+                            0.000564717163797468,
+                            0.0013301536673679948,
+                            0.0007393635460175574,
+                            0.0007767099887132645,
+                            0.0010797568829730153,
+                            0.0008857094217091799,
+                            0.0007383401389233768,
+                            0.0007181719993241131,
+                            0.0006278209621086717,
+                            0.000622601539362222,
+                            0.0006701787351630628,
+                            0.0005583862075582147,
+                            0.000769606966059655,
+                            0.0005976808024570346,
+                            0.0007832985138520598,
+                            0.0009059514850378036,
+                            0.000624670647084713,
+                            0.0011077580275014043,
+                            0.0007115105399861932,
+                            0.0014441285748034716,
+                            0.0006928564980626106,
+                            0.00140839081723243,
+                            0.0007134629413485527,
+                            0.0006863630260340869,
+                            0.0008639373118057847,
+                            0.0008925585425458848,
+                            0.0006539030582644045,
+                            0.0007157280342653394,
+                            0.0008994899690151215,
+                            0.0007541413069702685,
+                            0.0008779162890277803,
+                            0.0008626811904832721,
+                            0.0010930169373750687,
+                            0.0008360436768271029,
+                            0.000516226515173912,
+                            0.0008781215874478221,
+                            0.000740616989787668,
+                            0.0005550590576604009,
+                            0.0006354339420795441,
+                            0.0009714106563478708,
+                            0.0011319037294015288,
+                            0.000683894963003695,
+                            0.0008909969474188983,
+                            0.0005911845364607871,
+                            0.0007319003925658762,
+                            0.0012970161624252796,
+                            0.0006554731517098844,
+                            0.0009159381152130663,
+                            0.0008584765018895268,
+                            0.0006854198290966451,
+                            0.0005467599839903414,
+                            0.0012187025276944041,
+                            0.0014363252557814121,
+                            0.0005880362587049603,
+                            0.000683149672113359,
+                            0.0010465706000104547,
+                            0.00056100805522874,
+                            0.0008933335193432868,
+                            0.0011161939473822713,
+                            0.0006958229932934046,
+                            0.000764507451094687,
+                            0.0006480991141870618,
+                            0.0009360830299556255,
+                            0.0016949075507000089,
+                            0.0005776990437880158,
+                            0.0006234933971427381,
+                            0.0005400937516242266,
+                            0.0006452577654272318,
+                            0.0014096169034019113,
+                            0.0006374815129674971,
+                            0.0005717469030059874,
+                            0.0007126900600269437,
+                            0.0007876462186686695,
+                            0.0012578652240335941,
+                            0.0009484610636718571,
+                            0.0011827313574030995,
+                            0.000668706779833883,
+                            0.0008294347790069878,
+                            0.0007360386662185192,
+                            0.0010041218483820558,
+                            0.0007207145681604743,
+                            0.0007848090026527643,
+                            0.0005779435159638524,
+                            0.0010581969982013106,
+                            0.0008903833222575486,
+                            0.0008269764948636293,
+                            0.0005527908797375858,
+                            0.0006522084586322308,
+                            0.0006422698497772217,
+                            0.0007602217956446111,
+                            0.0005834586336277425,
+                            0.0006917366408742964,
+                            0.000762985902838409,
+                            0.000550927477888763,
+                            0.000514284591190517,
+                            0.0005663449410349131,
+                            0.0006444153259508312,
+                            0.0006815241067670286,
+                            0.0006396253011189401,
+                            0.0006720111705362797,
+                            0.0006796157103963196,
+                            0.0010758499847725034,
+                            0.0007280937279574573,
+                            0.0007034930749796331,
+                            0.0007429165416397154,
+                            0.0007403751369565725,
+                            0.0011673857225105166,
+                            0.0007197711383923888,
+                            0.0008353208540938795,
+                            0.0006450990913435817,
+                            0.0006726118735969067,
+                            0.0008339800406247377,
+                            0.0006890208460390568,
+                            0.0007040120544843376,
+                            0.0005709020188078284,
+                            0.0007551913149654865,
+                            0.0010945817921310663,
+                            0.0009060773300006986,
+                            0.0006313633057288826,
+                            0.0008336816099472344,
+                            0.00107828329782933,
+                            0.0008269865065813065,
+                            0.0006061131716705859,
+                            0.000679517223034054,
+                            0.0005572011577896774,
+                            0.000593576580286026,
+                            0.0005432001780718565,
+                            0.0005147097399458289,
+                            0.000738284841645509,
+                            0.0006278596119955182,
+                            0.0006123760249465704,
+                            0.0006270556477829814,
+                            0.000594191369600594,
+                            0.0009435354731976986,
+                            0.0006342537817545235,
+                            0.0008359157363884151,
+                            0.0005955586675554514,
+                            0.0006462207529693842,
+                            0.0007597882067784667,
+                            0.0005523308063857257,
+                            0.0007268107729032636,
+                            0.0006805264856666327,
+                            0.000592277676332742,
+                            0.0011626092018559575,
+                            0.0008114600786939263,
+                            0.0006261273520067334,
+                            0.0005917672533541918,
+                            0.0004922723164781928,
+                            0.0005758809274993837,
+                            0.0007588102016597986,
+                            0.0005325533566065133,
+                            0.000544448324944824,
+                            0.0006750347092747688,
+                            0.0006745659047737718,
+                            0.0005985954194329679,
+                            0.0005324320518411696,
+                            0.0012691962765529752,
+                            0.0005778578342869878,
+                            0.0005646629724651575,
+                            0.0006384123116731644,
+                            0.0006177001632750034,
+                            0.0006008626078255475,
+                            0.0007163551636040211,
+                            0.0005832131719216704,
+                            0.0006699307123199105,
+                            0.0005660074530169368,
+                            0.0007307425257749856,
+                            0.0006739477394148707,
+                            0.000850439362693578,
+                            0.000557838415261358,
+                            0.0005688642268069088,
+                            0.000729014805983752,
+                            0.0007913559093140066,
+                            0.000613609969150275,
+                            0.0006310503813438118,
+                            0.0006403826409950852,
+                            0.000728440994862467,
+                            0.0007013691356405616,
+                            0.000556433922611177,
+                            0.00055981200421229,
+                            0.0007454684819094837,
+                            0.0006642730440944433,
+                            0.0006291247555054724,
+                            0.0006423641461879015,
+                            0.0005180486477911472,
+                            0.0008884698618203402,
+                            0.0009593616123311222,
+                            0.0005486508016474545,
+                            0.0007425184594467282,
+                            0.0010013377759605646,
+                            0.0006216881447471678,
+                            0.0006737440708093345,
+                            0.000687172170728445,
+                            0.0006305872229859233,
+                            0.000772931904066354,
+                            0.0005204923218116164,
+                            0.0005536284297704697,
+                            0.0006286273710429668,
+                            0.0006829447811469436,
+                            0.0007972819730639458,
+                            0.0006316585931926966,
+                            0.0005424707778729498,
+                            0.0006129437824711204,
+                            0.0007027820684015751,
+                            0.0006490121013484895,
+                            0.000773098727222532,
+                            0.0006470077787525952,
+                            0.0010768427746370435,
+                            0.0005769976414740086,
+                            0.0009276527562178671,
+                            0.0005332485889084637,
+                            0.0005788410198874772,
+                            0.0005638208240270615,
+                            0.0005301434430293739,
+                            0.0008046958828344941,
+                            0.000577347120270133,
+                            0.000649122754111886,
+                            0.0007121404632925987,
+                            0.0006105340435169637,
+                            0.0005515039665624499,
+                            0.0005735561135224998,
+                            0.0006868687341921031,
+                            0.000691491411998868,
+                            0.0008096300298348069,
+                            0.0006082135951146483,
+                            0.0006024131434969604,
+                            0.0007752305245958269,
+                            0.0006205237586982548,
+                            0.0008950835326686502,
+                            0.0005728686228394508,
+                            0.0005651397514156997,
+                            0.0006246742559596896,
+                            0.0006012517842464149,
+                            0.0005855185445398092,
+                            0.0005148737691342831,
+                            0.0007559545338153839,
+                            0.0005945903831161559,
+                            0.0006447969935834408,
+                            0.0006383740692399442,
+                            0.0012222472578287125,
+                            0.0006742742261849344,
+                            0.0007169388118200004,
+                            0.000555266859009862,
+                            0.0006187552353367209,
+                            0.0005937375244684517,
+                            0.0005774764576926827,
+                            0.0007197826635092497,
+                            0.0005723288631998003,
+                            0.0007387143559753895,
+                            0.0005206909845583141,
+                            0.0005846255808137357,
+                            0.0006463719764724374,
+                            0.0005036827060393989,
+                            0.0007452253485098481,
+                            0.0007391468971036375,
+                            0.0005636036512441933,
+                            0.0005667336517944932,
+                            0.0006550943944603205,
+                            0.0006033493555150926,
+                            0.0008959001279436052,
+                            0.0006018694839440286,
+                            0.0006527471705339849,
+                            0.0004801899194717407,
+                            0.000737905444111675,
+                            0.0009369731415063143,
+                            0.0007613059715367854,
+                            0.0008413180476054549,
+                            0.000687639694660902,
+                            0.00048367399722337723,
+                            0.0006432650261558592,
+                            0.0004916259786114097,
+                            0.0006061920430511236,
+                            0.0005569335189647973,
+                            0.0006227167905308306,
+                            0.000520945992320776,
+                            0.0005191737436689436,
+                            0.0007404698408208787,
+                            0.0006312562618404627,
+                            0.000890162365976721,
+                            0.0005507280584424734,
+                            0.0005813623429276049,
+                            0.0006398269906640053,
+                            0.0006570158875547349,
+                            0.0005405295523814857,
+                            0.0005473866476677358,
+                            0.0006027328781783581,
+                            0.0006007055635564029,
+                            0.0007639795076102018,
+                            0.0005656029097735882,
+                            0.00047668800107203424,
+                            0.0009989271638914943,
+                            0.0006755326758138835,
+                            0.0005507044261321425,
+                            0.0005999872228130698,
+                            0.0005707130767405033,
+                            0.0005582327721640468,
+                            0.0005598630523309112,
+                            0.0005662726471200585,
+                            0.0005645622732117772,
+                            0.000595934281591326,
+                            0.0007517036283388734,
+                            0.0006594193400815129,
+                            0.0005570306093432009,
+                            0.0005580883007496595,
+                            0.0008394586038775742,
+                            0.0006888090283609927,
+                            0.0006445219623856246,
+                            0.001111985999159515,
+                            0.0006046222406439483,
+                            0.0006518873851746321,
+                            0.0007606859435327351,
+                            0.0007879894692450762,
+                            0.0006127840024419129,
+                            0.000674343085847795,
+                            0.0009457898559048772,
+                            0.000684860919136554,
+                            0.0005631993990391493,
+                            0.0005931147024966776,
+                            0.0007265444728545845,
+                            0.0006827272591181099,
+                            0.0008805154939182103,
+                            0.0006910691736266017,
+                            0.000500330759678036,
+                            0.000587180198635906,
+                            0.0005312205175869167,
+                            0.0006576976156793535,
+                            0.0009771069744601846,
+                            0.0006550479447469115,
+                            0.0006311634206213057,
+                            0.0006147528765723109,
+                            0.0007145465351641178,
+                            0.0006318740779533982,
+                            0.0011597113916650414,
+                            0.0006044813198968768,
+                            0.0005946058663539588,
+                            0.0006537307053804398,
+                            0.0006672117742709816,
+                            0.0005819381913170218,
+                            0.0009833687217906117,
+                            0.0012124154018238187,
+                            0.0005321134813129902,
+                            0.0005152715020813048,
+                            0.0006672085146419704,
+                            0.0006421491852961481,
+                            0.0005971382488496602,
+                            0.0007021070341579616,
+                            0.0006407054024748504,
+                            0.0005422679241746664,
+                            0.0005933122592978179,
+                            0.0005814824835397303,
+                            0.0005451926263049245,
+                            0.0006673306343145669,
+                            0.0005036393413320184,
+                            0.0006541665643453598,
+                            0.0006277360371313989,
+                            0.0008132587536238134,
+                            0.0006447699852287769,
+                            0.0006888652569614351,
+                            0.0005537865217775106,
+                            0.0005862591206096113,
+                            0.0005314177251420915,
+                            0.0007184006972238421,
+                            0.0007197259110398591,
+                            0.000783097988460213,
+                            0.0006312609184533358,
+                            0.0007308535859920084,
+                            0.0005574531387537718,
+                            0.0006371060153469443,
+                            0.0006617478793486953,
+                            0.0008398750214837492,
+                            0.0006944469059817493,
+                            0.0007637282833456993,
+                            0.0005686117219738662,
+                            0.0006866484764032066,
+                            0.0005456890794448555,
+                            0.0005090432241559029,
+                            0.0006408730405382812,
+                            0.0005436674691736698,
+                            0.000568436342291534,
+                            0.0005672118859365582,
+                            0.0006471819360740483,
+                            0.000625183223746717,
+                            0.0007168968440964818,
+                            0.0006110226386226714,
+                            0.0005519608384929597,
+                            0.0006477807182818651,
+                            0.0005361317889764905,
+                            0.0007636528462171555,
+                            0.0005919164977967739,
+                            0.0007032705470919609,
+                            0.0007171883480623364,
+                            0.0007315170369111001,
+                            0.0005551133072003722,
+                            0.0006828199257142842,
+                            0.00046276545617729425,
+                            0.0005686486256308854,
+                            0.0005825120024383068,
+                            0.000540176231879741,
+                            0.0005701617919839919,
+                            0.0005372220766730607,
+                            0.0007506022811867297,
+                            0.000643966079223901,
+                            0.000593349861446768,
+                            0.0006071995594538748,
+                            0.0005462639965116978,
+                            0.0006654211902059615,
+                            0.0006337781669571996,
+                            0.0005957235698588192,
+                            0.0006122886552475393,
+                            0.0007494971505366266,
+                            0.0006199039053171873,
+                            0.0006008818163536489,
+                            0.0005884157726541162,
+                            0.0006200784118846059,
+                            0.0007198834209702909,
+                            0.0006097452715039253,
+                            0.0005535203963518143,
+                            0.0006803791620768607,
+                            0.0006496531423181295,
+                            0.000569057825487107,
+                            0.0006286908173933625,
+                            0.0006766399019397795,
+                            0.0005575133254751563,
+                            0.0005503135034814477,
+                            0.0006059065344743431,
+                            0.0005662085022777319,
+                            0.0006491299718618393,
+                            0.0005882217083126307,
+                            0.000584616675041616,
+                            0.0006327323499135673,
+                            0.0006203179364092648,
+                            0.0005728048854507506,
+                            0.0005576873663812876,
+                            0.0005239066667854786,
+                            0.0005924638826400042,
+                            0.0005200799787417054,
+                            0.000546872615814209,
+                            0.0006253384635783732,
+                            0.0006111461552791297,
+                            0.0008026707218959928,
+                            0.0006623525405302644,
+                            0.0005258498131297529,
+                            0.0006606979295611382,
+                            0.0007059875642880797,
+                            0.0006177361938171089,
+                            0.0006287617143243551,
+                            0.000504870607983321,
+                            0.0007557804929092526,
+                            0.0007015843875706196,
+                            0.0007572268368676305,
+                            0.0006066125934012234,
+                            0.0005520973936654627,
+                            0.0007622907869517803,
+                            0.0005379684153012931,
+                            0.0005472971242852509,
+                            0.0007543053361587226,
+                            0.0005566162872128189,
+                            0.0006217931513674557,
+                            0.0006450994987972081,
+                            0.0006901451270096004,
+                            0.0005598490824922919,
+                            0.0005981047288514674,
+                            0.000630524882581085,
+                            0.0008167260093614459,
+                            0.0006130508845672011,
+                            0.0006238023051992059,
+                            0.0006387344328686595,
+                            0.0005563446902669966,
+                            0.0007436052546836436,
+                            0.0005702261114493012,
+                            0.000524317380040884,
+                            0.000677941890899092,
+                            0.00058619404444471,
+                            0.0005221057217568159,
+                            0.0005313993315212429,
+                            0.0005853588227182627,
+                            0.0008623817702755332,
+                            0.0006553838611580431,
+                            0.000720812997315079,
+                            0.0007252396899275482,
+                            0.0007010053959675133,
+                            0.0006560467882081866,
+                            0.0006386394379660487,
+                            0.0006061871536076069,
+                            0.000570095086004585,
+                            0.0005909604369662702,
+                            0.0005797648336738348,
+                            0.0005504952277988195,
+                            0.0005930638290010393,
+                            0.0007565930718556046,
+                            0.000619651866145432,
+                            0.0005471417680382729,
+                            0.0006916064885444939,
+                            0.0006421809084713459,
+                            0.0005808199057355523,
+                            0.000660967780277133,
+                            0.0004981582169421017,
+                            0.000611413037404418,
+                            0.0005897685186937451,
+                            0.0007470538839697838,
+                            0.0005097771645523608,
+                            0.0006329035386443138,
+                            0.0005335994064807892,
+                            0.0008377862977795303,
+                            0.0005817679921165109,
+                            0.0008113953517749906,
+                            0.0009010869543999434,
+                            0.0007375429850071669,
+                            0.0006136692245490849,
+                            0.0006475205882452428,
+                            0.0006746467552147806,
+                            0.0007969810976646841,
+                            0.0007342252065427601,
+                            0.0008209061343222857,
+                            0.0006557254819199443,
+                            0.0005921717965975404,
+                            0.0005920050316490233,
+                            0.0007181658875197172,
+                            0.0006330263568088412,
+                            0.0007819929160177708,
+                            0.0009835828095674515,
+                            0.0006143815699033439,
+                            0.0007251492934301496,
+                            0.0006874020909890532,
+                            0.0006477186689153314,
+                            0.000582225329708308,
+                            0.0010207198793068528,
+                            0.0009378535905852914,
+                            0.0007969973376020789,
+                            0.0008313162834383547,
+                            0.0008358543273061514,
+                            0.0005851518944837153,
+                            0.0007417557644657791,
+                            0.0008545606397092342,
+                            0.0006504061748273671,
+                            0.0005945060984231532,
+                            0.0006320231477729976,
+                            0.0006182072684168816,
+                            0.0006246961420401931,
+                            0.0009621934150345623,
+                            0.00062438944587484,
+                            0.0006114019779488444,
+                            0.0006726552965119481,
+                            0.0006173820584081113,
+                            0.0007275649113580585,
+                            0.0006484753685072064,
+                            0.0005487511516548693,
+                            0.0006732696201652288,
+                            0.0006963533232919872,
+                            0.0008412761380895972,
+                            0.0006088290247134864,
+                            0.0008585074683651328,
+                            0.0007026453968137503,
+                            0.0006019805441610515,
+                            0.0006206904072314501,
+                            0.0007092574378475547,
+                            0.0005785642424598336,
+                            0.0006166717503219843,
+                            0.0005616816342808306,
+                            0.000836056424304843,
+                            0.0006721456884406507,
+                            0.0006862392765469849,
+                            0.0008495430811308324,
+                            0.0006332737393677235,
+                            0.0005156188271939754,
+                            0.0006561867776326835,
+                            0.0006387534085661173,
+                            0.0006579184555448592,
+                            0.0008084276923909783,
+                            0.0007458609179593623,
+                            0.000589611241593957,
+                            0.0006143015925772488,
+                            0.0006761534605175257,
+                            0.0005991845391690731,
+                            0.0007320405566133559,
+                            0.0006293419282883406,
+                            0.0005678380257450044,
+                            0.0006562837515957654,
+                            0.0008678989252075553,
+                            0.0005985449533909559,
+                            0.000803945236839354,
+                            0.0005894121713936329,
+                            0.0005824590334668756,
+                            0.0006772060878574848,
+                            0.0005827242275699973,
+                            0.0009193331352435052,
+                            0.0005363077507354319,
+                            0.000713514571543783,
+                            0.0006345948786474764,
+                            0.000893545220606029,
+                            0.0005438870284706354,
+                            0.0006058997241780162,
+                            0.000613673881161958,
+                            0.0005793679738417268,
+                            0.0005222582840360701,
+                            0.0009322981350123882,
+                            0.0005420022644102573,
+                            0.000544815615285188,
+                            0.0005666238721460104,
+                            0.0006570888217538595,
+                            0.0007085592369548976,
+                            0.0006814402877353132,
+                            0.0006344934226945043,
+                            0.0006078872247599065,
+                            0.0008871497702784836,
+                            0.0006095375865697861,
+                            0.0006291494937613606,
+                            0.0011375857284292579,
+                            0.0006020694854669273,
+                            0.0006416275864467025,
+                            0.0005782137159258127,
+                            0.0006262938841246068,
+                            0.0007878709002397954,
+                            0.0006901964079588652,
+                            0.0007265078602358699,
+                            0.0007581388344988227,
+                            0.0006602671346627176,
+                            0.0006795873050577939,
+                            0.0006495965644717216,
+                            0.000788042088970542,
+                            0.0008886340074241161,
+                            0.0007655051304027438,
+                            0.0007100966759026051,
+                            0.0005581495352089405,
+                            0.0007468812982551754,
+                            0.0005595209659077227,
+                            0.000679665885400027,
+                            0.0006160960183478892,
+                            0.0009292743052355945,
+                            0.0006303631234914064,
+                            0.0006482001044787467,
+                            0.0006890007643960416,
+                            0.0005993673694320023,
+                            0.0006079585873521864,
+                            0.0008503533317707479,
+                            0.0006889650830999017,
+                            0.000662204169202596,
+                            0.0008100937702693045,
+                            0.0006062078173272312,
+                            0.0007087685517035425,
+                            0.0005918850656598806,
+                            0.0006015513790771365,
+                            0.0008023210102692246,
+                            0.0007146619027480483,
+                            0.0006862720474600792,
+                            0.0006278773653320968,
+                            0.0006402545259334147,
+                            0.0006547208759002388,
+                            0.0006410845089703798,
+                            0.0006355689838528633,
+                            0.0005842955433763564,
+                            0.0006653723539784551,
+                            0.0008863803232088685,
+                            0.0007467243704013526,
+                            0.000554337864741683,
+                            0.0005676386062987149,
+                            0.0006314080092124641,
+                            0.0005756262107752264,
+                            0.0006574190920218825,
+                            0.0006334256613627076,
+                            0.0006057383143343031,
+                            0.0007229366456158459,
+                            0.0005932078929618001,
+                            0.0005529714981094003,
+                            0.0005991176585666835,
+                            0.0007219401886686683,
+                            0.0005163438618183136,
+                            0.0006456844857893884,
+                            0.0006977058365009725,
+                            0.0005909207975491881,
+                            0.0006095382268540561,
+                            0.0006619502091780305,
+                            0.0007037695031613111,
+                            0.0005794755998067558,
+                            0.0006926365313120186,
+                            0.0005796024342998862,
+                            0.0007222766289487481,
+                            0.0007132759783416986,
+                            0.0006414917879737914,
+                            0.0007177404477261007,
+                            0.0005461148684844375,
+                            0.0006989846588112414,
+                            0.0008683954365551472,
+                            0.000723662378732115,
+                            0.000699433614499867,
+                            0.0008520899573341012,
+                            0.0007021752535365522,
+                            0.0006684939726255834,
+                            0.0007674561347812414,
+                            0.0005757965845987201,
+                            0.0006622446817345917,
+                            0.0006136280135251582,
+                            0.000661009456962347,
+                            0.0006009233184158802,
+                            0.0005560613935813308,
+                            0.0006197468028403819,
+                            0.0006279345834627748,
+                            0.0007679800619371235,
+                            0.0006352384807541966,
+                            0.000629441172350198,
+                            0.0006857174448668957,
+                            0.0006703567341901362,
+                            0.0006488467915914953,
+                            0.0006690845475532115,
+                            0.0006282163085415959,
+                            0.0005599010619334877,
+                            0.0006753151537850499,
+                            0.0005396789056248963,
+                            0.0006092609837651253,
+                            0.0006066432688385248,
+                            0.0006375382072292268,
+                            0.0006194448214955628,
+                            0.0005826871492899954,
+                            0.0007320921868085861,
+                            0.0005968904006294906,
+                            0.0006197344046086073,
+                            0.0005172601668164134,
+                            0.0005571233341470361,
+                            0.0006639168132096529,
+                            0.0005544660380110145,
+                            0.0006302105030044913,
+                            0.0007459839689545333,
+                            0.0006883783498778939,
+                            0.0005169441574253142,
+                            0.0006602128851227462,
+                            0.0007416884764097631,
+                            0.0006456319242715836,
+                            0.0008493505883961916,
+                            0.0007643919088877738,
+                            0.0005149389035068452,
+                            0.0006075875135138631,
+                            0.0007240985869430006,
+                            0.0005929574253968894,
+                            0.0006517800502479076,
+                            0.0006130984984338284,
+                            0.0007211214397102594,
+                            0.0006428798078559339,
+                            0.0005574083770625293,
+                            0.0006079858867451549,
+                            0.0005921812262386084,
+                            0.0006242812960408628,
+                            0.0006857119151391089,
+                            0.0007012012647464871,
+                            0.0006174636073410511,
+                            0.0006822910509072244,
+                            0.0006673571770079434,
+                            0.0006733520422130823,
+                            0.0006310363532975316,
+                            0.0006559529574587941,
+                            0.0007443715585395694,
+                            0.000587648362852633,
+                            0.0005110971978865564,
+                            0.0010027821408584714,
+                            0.0005497313686646521,
+                            0.0005770972929894924,
+                            0.0007338373688980937,
+                            0.0006795550580136478,
+                            0.0005093818763270974,
+                            0.0005668951198458672,
+                            0.0005450149765238166,
+                            0.0006529578822664917,
+                            0.0007003498030826449,
+                            0.000550287077203393,
+                            0.000527967291418463,
+                            0.0006644694949500263,
+                            0.0006468340288847685,
+                            0.0005548092303797603,
+                            0.0006104019703343511,
+                            0.0006462986930273473,
+                            0.0005267751985229552,
+                            0.0006925230263732374,
+                            0.0005910705076530576,
+                            0.0006794581422582269,
+                            0.0005477224476635456,
+                            0.0006268659490160644,
+                            0.0005526675377041101,
+                            0.0006017805426381528,
+                            0.0006159000913612545,
+                            0.0005755956517532468,
+                            0.0006444669561460614,
+                            0.0006990348920226097,
+                            0.0006156713352538645,
+                            0.000542946218047291,
+                            0.0007227763417176902,
+                            0.0006520944880321622,
+                            0.0005556825781241059,
+                            0.0006452113157138228,
+                            0.0006485223420895636,
+                            0.0005838431534357369,
+                            0.0004980380763299763,
+                            0.0005829663714393973,
+                            0.0007468067924492061,
+                            0.0007592176552861929,
+                            0.0006590557168237865,
+                            0.0006209448329173028,
+                            0.0006230580038391054,
+                            0.0006388388574123383,
+                            0.0005943751893937588,
+                            0.0007372485706582665,
+                            0.0005618822178803384,
+                            0.0006306037539616227,
+                            0.0006640289211645722,
+                            0.0005659349844790995,
+                            0.0005918153910897672,
+                            0.0006303361733444035,
+                            0.0005438750959001482,
+                            0.0005794658791273832,
+                            0.0005976242246106267,
+                            0.0005294805741868913,
+                            0.0005316967144608498,
+                            0.0005926762823946774,
+                            0.0005207752110436559,
+                            0.0005457202205434442,
+                            0.0006187158287502825,
+                            0.0007065136451274157,
+                            0.0006255038897506893,
+                            0.0005096595850773156,
+                            0.0005487445741891861,
+                            0.0006754372152499855,
+                            0.0006380123668350279,
+                            0.0005928824539296329,
+                            0.0006346750888042152,
+                            0.0007100141956470907,
+                            0.0006369980401359499,
+                            0.0006321428809314966,
+                            0.0006116176955401897,
+                            0.0005424369010142982,
+                            0.0005745937232859433,
+                            0.000621867657173425,
+                            0.0007595068891532719,
+                            0.0006953570991754532,
+                            0.0005651105893775821,
+                            0.0006374877993948758,
+                            0.0005523258587345481,
+                            0.0006242282106541097,
+                            0.0005580576835200191,
+                            0.0007940769428387284,
+                            0.0005214600241743028,
+                            0.0005349195562303066,
+                            0.0007437568856403232,
+                            0.0005397532368078828,
+                            0.0006315007922239602,
+                            0.000595570367295295,
+                            0.0005169243086129427,
+                            0.0006284248665906489,
+                            0.0006009074277244508,
+                            0.0005686567164957523,
+                            0.0005382320960052311,
+                            0.0005989981000311673,
+                            0.0006080101011320949,
+                            0.0006998936296440661,
+                            0.0006327042356133461,
+                            0.000558163330424577,
+                            0.0005475354264490306,
+                            0.0007799883605912328,
+                            0.0006071659736335278,
+                            0.0005820157821290195,
+                            0.0005543230217881501,
+                            0.0006092888070270419,
+                            0.0006641779327765107,
+                            0.0006689513684250414,
+                            0.0005655122804455459,
+                            0.0008564328891225159,
+                            0.0005211624666117132,
+                            0.0006046960479579866,
+                            0.0006239901995286345,
+                            0.0005884439451619983,
+                            0.0007448472315445542,
+                            0.0005461171385832131,
+                            0.0009647133992984891,
+                            0.0007486167596653104,
+                            0.0006785386358387768,
+                            0.00048817836795933545,
+                            0.0006280745728872716,
+                            0.0005058757378719747,
+                            0.0006318764062598348,
+                            0.0006646681576967239,
+                            0.0006181859062053263,
+                            0.0006625090609304607,
+                            0.0005010801833122969,
+                            0.0006928040529601276,
+                            0.0007248527836054564,
+                            0.0005927021848037839,
+                            0.000683398509863764,
+                            0.000638761033769697,
+                            0.0008137088152579963,
+                            0.000516678555868566,
+                            0.0005881194374524057,
+                            0.0006125482614152133,
+                            0.0005178333376534283,
+                            0.0005100604030303657,
+                            0.00046913340338505805,
+                            0.0005529780173674226,
+                            0.0005964106530882418,
+                            0.0005969784106127918,
+                            0.0006043621688149869,
+                            0.0005407983553595841,
+                            0.0006072743562981486,
+                            0.0005823398823849857,
+                            0.0005271268892101943,
+                            0.0005608050851151347,
+                            0.0005704621435143054,
+                            0.0007130607846193016,
+                            0.0005969507619738579,
+                            0.0006686192937195301,
+                            0.0006026666960678995,
+                            0.0007553088362328708,
+                            0.0006721205427311361,
+                            0.0006434883689507842,
+                            0.0006762862321920693,
+                            0.0007828549132682383,
+                            0.000613431038800627,
+                            0.0006200391799211502,
+                            0.0006604601512663066,
+                            0.0006395088857971132,
+                            0.0005377148045226932,
+                            0.0005241477629169822,
+                            0.0005720430053770542,
+                            0.0006566362571902573,
+                            0.0005715642473660409,
+                            0.0006585851078853011,
+                            0.0007867890526540577,
+                            0.0006360132247209549,
+                            0.0005709936958737671,
+                            0.0006538694142363966,
+                            0.0005633109249174595,
+                            0.0006047733477316797,
+                            0.0005844393745064735,
+                            0.0005966289900243282,
+                            0.0007060181233100593,
+                            0.0005769199342466891,
+                            0.0008653565309941769,
+                            0.0007773499237373471,
+                            0.0007088436977937818,
+                            0.0007594813941977918,
+                            0.0006980170146562159,
+                            0.0005554796080105007,
+                            0.0007254299707710743,
+                            0.0006307056173682213,
+                            0.0008675612043589354,
+                            0.0007619595271535218,
+                            0.0007114626350812614,
+                            0.0006252569728530943,
+                            0.000586893642321229,
+                            0.0005688121309503913,
+                            0.0005341165815480053,
+                            0.0006738046649843454,
+                            0.0006585263181477785,
+                            0.0005299176555126905,
+                            0.0005512564675882459,
+                            0.0005232691182754934,
+                            0.0006070827948860824,
+                            0.0005259590689092875,
+                            0.0007304133614525199,
+                            0.0006474212277680635,
+                            0.0006542205228470266,
+                            0.0005833034520037472,
+                            0.0005474757635965943,
+                            0.0007934953318908811,
+                            0.0006865913746878505,
+                            0.0006647247937507927,
+                            0.0005902407574467361,
+                            0.000603671942371875,
+                            0.000754066277295351,
+                            0.0006065298803150654,
+                            0.0005228602676652372,
+                            0.0005023572593927383,
+                            0.0006085631321184337,
+                            0.000731237989384681,
+                            0.000730940664652735,
+                            0.000667446234729141,
+                            0.0005568954511545599,
+                            0.0006723082042299211,
+                            0.0005203897599130869,
+                            0.0006971704424358904,
+                            0.0005922441487200558,
+                            0.0006202365620993078,
+                            0.0013087649131193757,
+                            0.0006507299840450287,
+                            0.0006705549894832075,
+                            0.0005891447071917355,
+                            0.0005960718845017254,
+                            0.0006014917162247002,
+                            0.0006787978927604854,
+                            0.0006729463930241764,
+                            0.0007109337602742016,
+                            0.0007372836698777974,
+                            0.0007851074915379286,
+                            0.0005611265078186989,
+                            0.0006068894290365279,
+                            0.0006239525391720235,
+                            0.0007898014737293124,
+                            0.000543034344445914,
+                            0.0007792949909344316,
+                            0.0006563572096638381,
+                            0.0006400396232493222,
+                            0.0005241398466750979,
+                            0.0006917184218764305,
+                            0.0006397450342774391,
+                            0.0005176628474146128,
+                            0.0006335983634926379,
+                            0.000627105706371367,
+                            0.0005097255343571305,
+                            0.0006611882126890123,
+                            0.0007317255949601531,
+                            0.0007467850227840245,
+                            0.0005924266297370195,
+                            0.0007386929355561733,
+                            0.0008010466699488461,
+                            0.0008064661524258554,
+                            0.0006450896617025137,
+                            0.0005401486996561289,
+                            0.0006678627105429769,
+                            0.0005868417792953551,
+                            0.0005783340311609209,
+                            0.0005674677668139338,
+                            0.0005717304884456098,
+                            0.0007350020459853113,
+                            0.0006360379047691822,
+                            0.0005209912778809667,
+                            0.0008813802851364017,
+                            0.0006274270708672702,
+                            0.0006644364329986274,
+                            0.0008224160992540419,
+                            0.0006237621419131756,
+                            0.0005913901259191334,
+                            0.0006699358928017318,
+                            0.0007186693255789578,
+                            0.0005913568893447518,
+                            0.0006589633412659168,
+                            0.0005833387258462608,
+                            0.000560714106541127,
+                            0.0006266843411140144,
+                            0.0011464363196864724,
+                            0.0007727050688117743,
+                            0.0007690396159887314,
+                            0.0007313925307244062,
+                            0.0006002081208862364,
+                            0.0005419442895799875,
+                            0.0006248095887713134,
+                            0.0005705531220883131,
+                            0.0006060181767679751,
+                            0.000756581372115761,
+                            0.0005530838971026242,
+                            0.0005713714635930955,
+                            0.000659339944832027,
+                            0.0007460910710506141,
+                            0.000544324517250061,
+                            0.000660953635815531,
+                            0.0006606591050513089,
+                            0.0006812226492911577,
+                            0.0006372242933139205,
+                            0.0006218962371349335,
+                            0.0007000574260018766,
+                            0.0005323518998920918,
+                            0.0007204676512628794,
+                            0.0006928738439455628,
+                            0.0007460928754881024,
+                            0.0006353824865072966,
+                            0.0006142883212305605,
+                            0.0005481266416609287,
+                            0.0006413076189346611,
+                            0.0006166533567011356,
+                            0.0005924772704020143,
+                            0.0006905986228957772,
+                            0.0005674015264958143,
+                            0.0007380073075182736,
+                            0.0006092466646805406,
+                            0.0005774795426987112,
+                            0.000601057312451303,
+                            0.000654224946629256,
+                            0.0007220205734483898,
+                            0.0006140329060144722,
+                            0.0006790393381379545,
+                            0.0006037771818228066,
+                            0.0006253314786590636,
+                            0.0005996705731377006,
+                            0.0006224082317203283,
+                            0.0006064347107894719,
+                            0.0005610029911622405,
+                            0.0008560648420825601,
+                            0.0005446495488286018,
+                            0.0005409216391853988,
+                            0.000603420368861407,
+                            0.0006346331792883575,
+                            0.0007360212621279061,
+                            0.0006981163169257343,
+                            0.0005698170862160623,
+                            0.0007237884565256536,
+                            0.0007428756216540933,
+                            0.0007164674461819232,
+                            0.0006713782204315066,
+                            0.0005415158229880035,
+                            0.0006532514235004783,
+                            0.0006225731922313571,
+                            0.0005917802918702364,
+                            0.000536322477273643,
+                            0.0006217582267709076,
+                            0.0005227776127867401,
+                            0.0007500626379624009,
+                            0.0006529017118737102,
+                            0.0005332191940397024,
+                            0.0005968771874904633,
+                            0.0005594775429926813,
+                            0.0006257757777348161,
+                            0.000733576191123575,
+                            0.000863286666572094,
+                            0.0005400420632213354,
+                            0.00078766536898911,
+                            0.0006211904692463577,
+                            0.0005728879477828741,
+                            0.0006038981373421848,
+                            0.0006392436916939914,
+                            0.000591639953199774,
+                            0.0005439955275505781,
+                            0.0006447962950915098,
+                            0.0008237460860982537,
+                            0.0005351164727471769,
+                            0.0005938855465501547,
+                            0.0005169385694898665,
+                            0.0005747172399424016,
+                            0.0005718100001104176,
+                            0.000644276849925518,
+                            0.0007050684071145952,
+                            0.0007033580332063138,
+                            0.00064986472716555,
+                            0.0006949067465029657,
+                            0.0007159746019169688,
+                            0.0006152035784907639,
+                            0.0006685847765766084,
+                            0.0005874757771380246,
+                            0.0005244703497737646,
+                            0.0005894678761251271,
+                            0.0006266747950576246,
+                            0.0006707679131068289,
+                            0.0006112923147156835,
+                            0.0006668673013336957,
+                            0.0005472474149428308,
+                            0.0005627952050417662,
+                            0.0005998231936246157,
+                            0.0006082071922719479,
+                            0.0006977938464842737,
+                            0.0005917717353440821,
+                            0.0007059365743771195,
+                            0.000579698127694428,
+                            0.0006966815562918782,
+                            0.0007605588180013001,
+                            0.0006173434667289257,
+                            0.0006366870366036892,
+                            0.0006169591215439141,
+                            0.0006187630933709443,
+                            0.0005075680091977119,
+                            0.0008310870616696775,
+                            0.000604099128395319,
+                            0.0006268442957662046,
+                            0.0006469835643656552,
+                            0.0007185455760918558,
+                            0.0005920644616708159,
+                            0.0006799621041864157,
+                            0.0005706852534785867,
+                            0.0006047557108104229,
+                            0.0005835352349095047,
+                            0.0005321823409758508,
+                            0.0006348539609462023,
+                            0.0005134774255566299,
+                            0.0005658558802679181,
+                            0.0005770988645963371,
+                            0.000670403300318867,
+                            0.0006408999906852841,
+                            0.0006099757156334817,
+                            0.0005783644155599177,
+                            0.0006168947438709438,
+                            0.0006622663349844515,
+                            0.0008794443565420806,
+                            0.0008160287397913635,
+                            0.0007831510156393051,
+                            0.00066590984351933,
+                            0.0006284560658968985,
+                            0.0005678160232491791,
+                            0.0006086738430894911,
+                            0.0005892490735277534,
+                            0.0006211344734765589,
+                            0.0005562498699873686,
+                            0.0005765366950072348,
+                            0.0005852727335877717,
+                            0.0006004859460517764,
+                            0.0006225755787454545,
+                            0.0005466713919304311,
+                            0.0005831577582284808,
+                            0.0005722683854401112,
+                            0.0006364323198795319,
+                            0.0006540808244608343,
+                            0.0006078769802115858,
+                            0.0005936852539889514,
+                            0.0006406872998923063,
+                            0.0006255554035305977,
+                            0.0007287219050340354,
+                            0.0005079077673144639,
+                            0.0006603060755878687,
+                            0.0006171694840304554,
+                            0.000556948478333652,
+                            0.0005442120018415153,
+                            0.0006982585182413459,
+                            0.0005694260471500456,
+                            0.0005525204469449818,
+                            0.0006931971292942762,
+                            0.0005609345971606672,
+                            0.0006791583145968616,
+                            0.0005581693258136511,
+                            0.0005749014089815319,
+                            0.0005887943552806973,
+                            0.0005319620831869543,
+                            0.0006460447330027819,
+                            0.0005484281573444605,
+                            0.0008205288322642446,
+                            0.0006141217891126871,
+                            0.0006257143686525524,
+                            0.000650314730592072,
+                            0.0006292061298154294,
+                            0.000624363252427429,
+                            0.0005655281711369753,
+                            0.0005721859633922577,
+                            0.0006481253076344728,
+                            0.0005808077985420823,
+                            0.0006343139102682471,
+                            0.000719082192517817,
+                            0.0005362226511351764,
+                            0.0006016592378728092,
+                            0.0007635697838850319,
+                            0.0006627370603382587,
+                            0.0007330312510021031,
+                            0.0005607641651295125,
+                            0.0005667505320161581,
+                            0.0005816120537929237,
+                            0.0005188394570723176,
+                            0.0005857077776454389,
+                            0.0006615984602831304,
+                            0.0006520207971334457,
+                            0.0007092810119502246,
+                            0.0006484166369773448,
+                            0.000588533584959805,
+                            0.0005804265383630991,
+                            0.0005488108145073056,
+                            0.0005585116450674832,
+                            0.0007005339721217752,
+                            0.0005940452101640403,
+                            0.0007237467798404396,
+                            0.0004923642845824361,
+                            0.0006239449139684439,
+                            0.0006866886396892369,
+                            0.0004895477904938161,
+                            0.0005158788990229368,
+                            0.0006081952014937997,
+                            0.0005776453763246536,
+                            0.0006235036416910589,
+                            0.000656745454762131,
+                            0.0006020539440214634,
+                            0.0005595211405307055,
+                            0.000542414840310812,
+                            0.0007300909492187202,
+                            0.0005051267798990011,
+                            0.0005245068459771574,
+                            0.0006985111976973712,
+                            0.0005784141831099987,
+                            0.0007604340789839625,
+                            0.0005822537350468338,
+                            0.0006349377799779177,
+                            0.00050885952077806,
+                            0.0006473531248047948,
+                            0.0005259576137177646,
+                            0.0005490256589837372,
+                            0.0006074132979847491,
+                            0.0005825308617204428,
+                            0.0006059144507162273,
+                            0.0006054056575521827,
+                            0.0006270201993174851,
+                            0.0005497107049450278,
+                            0.0006247066194191575,
+                            0.0005940224509686232,
+                            0.000784030242357403,
+                            0.0006671611336059868,
+                            0.0006089215166866779,
+                            0.0006114851566962898,
+                            0.0006320131360553205,
+                            0.0005898195668123662,
+                            0.0006945270579308271,
+                            0.0006064497283659875,
+                            0.0005731403362005949,
+                            0.000617357378359884,
+                            0.0006244780379347503,
+                            0.0005632514948956668,
+                            0.0005875441711395979,
+                            0.0005789388669654727,
+                            0.00059342390159145,
+                            0.0005565526080317795,
+                            0.0005873911431990564,
+                            0.0006357888341881335,
+                            0.0006227928097359836,
+                            0.0006645332905463874,
+                            0.0005691209807991982,
+                            0.0005329442792572081,
+                            0.0006595836021006107,
+                            0.0005206693313084543,
+                            0.0005190044757910073,
+                            0.0005578136770054698,
+                            0.0006245474214665592,
+                            0.0005695365252904594,
+                            0.000563720939680934,
+                            0.000687938358169049,
+                            0.0006136939045973122,
+                            0.0007939098868519068,
+                            0.0006385388551279902,
+                            0.0006298675434663892,
+                            0.000621984014287591,
+                            0.0006259538931772113,
+                            0.0005944454460404813,
+                            0.000707817729562521,
+                            0.0006202589720487595,
+                            0.0007515490287914872,
+                            0.0006276122876442969,
+                            0.0006424639141187072,
+                            0.0005131667712703347,
+                            0.0006872651865705848,
+                            0.0005672315019182861,
+                            0.0005294974544085562,
+                            0.0007092473679222167,
+                            0.0006024314789101481,
+                            0.0005819568177685142,
+                            0.0005396988708525896,
+                            0.0006596733001060784,
+                            0.0005700948531739414,
+                            0.0005112611688673496,
+                            0.0006578715983778238,
+                            0.0007538786740042269,
+                            0.0006195599562488496,
+                            0.0008041331311687827,
+                            0.0006174712325446308,
+                            0.0006732715992256999,
+                            0.0006723070400767028,
+                            0.0006192278233356774,
+                            0.000627279921900481,
+                            0.000566226604860276,
+                            0.0005527070607058704,
+                            0.0006322170956991613,
+                            0.0006300591048784554,
+                            0.0006912128301337361,
+                            0.0007127143326215446,
+                            0.0006787902093492448,
+                            0.0005671399994753301,
+                            0.0006419036653824151,
+                            0.0006460128934122622,
+                            0.0006440407014451921,
+                            0.0005665854550898075,
+                            0.0005213309777900577,
+                            0.0006403169827535748,
+                            0.0006547908415086567,
+                            0.0007112842868082225,
+                            0.000676624767947942,
+                            0.0005811983137391508,
+                            0.0005917891394346952,
+                            0.0006087670917622745,
+                            0.0005398254143074155,
+                            0.0007475580787286162,
+                            0.0006091000977903605,
+                            0.0006545981159433722,
+                            0.0006034785183146596,
+                            0.0006326871225610375,
+                            0.0006333844503387809,
+                            0.0006216716137714684,
+                            0.000625857908744365,
+                            0.0006871556397527456,
+                            0.0005757224280387163,
+                            0.0005264453357085586,
+                            0.0005202891770750284,
+                            0.0005620294832624495,
+                            0.0006245658732950687,
+                            0.000718875729944557,
+                            0.0005530191701836884,
+                            0.0006908020586706698,
+                            0.0007422881899401546,
+                            0.00065610371530056,
+                            0.000688373576849699,
+                            0.0005854074843227863,
+                            0.0006378222024068236,
+                            0.0005836129421368241,
+                            0.0006375401862896979,
+                            0.0006183769437484443,
+                            0.0006177405593916774,
+                            0.0005924075376242399,
+                            0.0007564227562397718,
+                            0.0006082805339246988,
+                            0.0006492417305707932,
+                            0.0007150766323320568,
+                            0.0005844723200425506,
+                            0.0006034642574377358,
+                            0.0006187730468809605,
+                            0.0005183712928555906,
+                            0.000826018862426281,
+                            0.0006545435171574354,
+                            0.000561754044611007,
+                            0.0005612666136585176,
+                            0.000501211907248944,
+                            0.0007530718576163054,
+                            0.000539960281457752,
+                            0.0006951534887775779,
+                            0.0005454798811115324,
+                            0.0005599043797701597,
+                            0.0006135646835900843,
+                            0.0005559232085943222,
+                            0.0006891518132761121,
+                            0.0005364370299503207,
+                            0.0006920784362591803,
+                            0.0005652543623000383,
+                            0.000572230841498822,
+                            0.0005579657736234367,
+                            0.0006814564112573862,
+                            0.0006909852963872254,
+                            0.0005414439947344363,
+                            0.0007561394013464451,
+                            0.0007103655952960253,
+                            0.0006039873696863651,
+                            0.0005082839634269476,
+                            0.0007193060009740293,
+                            0.0006666432018391788,
+                            0.0006909622461535037,
+                            0.0005474923527799547,
+                            0.0005721517372876406,
+                            0.0006251019076444209,
+                            0.0005928389145992696,
+                            0.0007346558850258589,
+                            0.0006288410513661802,
+                            0.0005723973736166954,
+                            0.0005594572867266834,
+                            0.0006620081840083003,
+                            0.0005728936521336436,
+                            0.0005893960478715599,
+                            0.0005465402500703931,
+                            0.0004949384019710124,
+                            0.0006720352685078979,
+                            0.0006341760163195431,
+                            0.000549273332580924,
+                            0.000615398574154824,
+                            0.000588390976190567,
+                            0.0007809223025105894,
+                            0.0007999524241313338,
+                            0.0006585223600268364,
+                            0.0005706633673980832,
+                            0.0006158381584100425,
+                            0.0006484644836746156,
+                            0.00064664933597669,
+                            0.0005399544024839997,
+                            0.0007092719315551221,
+                            0.0005590401124209166,
+                            0.0005993626546114683,
+                            0.000543775677215308,
+                            0.0005990209756419063,
+                            0.0005552515503950417,
+                            0.0005333877634257078,
+                            0.0006506689824163914,
+                            0.000540994165930897,
+                            0.0005885281134396791,
+                            0.0005803167587146163,
+                            0.0005444138660095632,
+                            0.0005369161954149604,
+                            0.0005626109195873141,
+                            0.0006649806746281683,
+                            0.0006329815951175988,
+                            0.0007686431054025888,
+                            0.0007355473935604095,
+                            0.0005688563687726855,
+                            0.0005682283081114292,
+                            0.000664594816043973,
+                            0.0005737179890275002,
+                            0.0007013790891505778,
+                            0.0006100991740822792,
+                            0.0005443184054456651,
+                            0.0005649158265441656,
+                            0.0004940209910273552,
+                            0.0005340362549759448,
+                            0.000507219519931823,
+                            0.0005971697391942143,
+                            0.0005832865135744214,
+                            0.0006990131223574281,
+                            0.000757901871111244,
+                            0.0005376431508921087,
+                            0.000549959484487772,
+                            0.0005514328368008137,
+                            0.0005380954244174063,
+                            0.0006952779949642718,
+                            0.0005673090345226228,
+                            0.0005412710597738624,
+                            0.0005729427793994546,
+                            0.0005649153608828783,
+                            0.0006017936975695193,
+                            0.0005587168270722032,
+                            0.0005846677231602371,
+                            0.0006187753169797361,
+                            0.0006247407291084528,
+                            0.000583830988034606,
+                            0.0006377894897013903,
+                            0.0006696200580336154,
+                            0.0006125234649516642,
+                            0.0006919853622093797,
+                            0.0004985714331269264,
+                            0.0005802556988783181,
+                            0.0007292241789400578,
+                            0.000595547491684556,
+                            0.0005904319696128368,
+                            0.0006227899575605989,
+                            0.0004732060479000211,
+                            0.0006429392378777266,
+                            0.0005646645440720022,
+                            0.0006403387524187565,
+                            0.0005429555312730372,
+                            0.0005482886335812509,
+                            0.0006636565085500479,
+                            0.0005738877225667238,
+                            0.0006105386419221759,
+                            0.0005449248128570616,
+                            0.0007329650688916445,
+                            0.0006206043181009591,
+                            0.0005433689220808446,
+                            0.0006476548733189702,
+                            0.0005760183557868004,
+                            0.0006534749991260469,
+                            0.0005710483528673649,
+                            0.0005950190825387836,
+                            0.0006638711784034967,
+                            0.0006397415418177843,
+                            0.0005851566093042493,
+                            0.0006342206615954638,
+                            0.0006067696958780289,
+                            0.0007781196036376059,
+                            0.0006311579491011798,
+                            0.0005640555173158646,
+                            0.0005772665608674288,
+                            0.0005979534471407533,
+                            0.0006015817634761333,
+                            0.0006764305871911347,
+                            0.0006145274383015931,
+                            0.0007005999214015901,
+                            0.0005805494729429483,
+                            0.0005557746044360101,
+                            0.0008169926586560905,
+                            0.0005541006685234606,
+                            0.0005807718844152987,
+                            0.0006633912562392652,
+                            0.0006231997977010906,
+                            0.000619924976490438,
+                            0.0007161461981013417,
+                            0.0005947641329839826,
+                            0.000616507779341191,
+                            0.0005477900849655271,
+                            0.0007018976612016559,
+                            0.0007111945888027549,
+                            0.000696966249961406,
+                            0.000565387075766921,
+                            0.0006329677416943014,
+                            0.0005619106232188642,
+                            0.0005991947837173939,
+                            0.000723479432053864,
+                            0.0006863742019049823,
+                            0.0006730116438120604,
+                            0.0006181247299537063,
+                            0.0006803942960686982,
+                            0.0006726732826791704,
+                            0.0005100535927340388,
+                            0.0005796427722088993,
+                            0.0007963879615999758,
+                            0.0006069301161915064,
+                            0.0007724713650532067,
+                            0.00060082454001531,
+                            0.000704375677742064,
+                            0.0007047069375403225,
+                            0.0005908522289246321,
+                            0.0006148968823254108,
+                            0.0007450199918821454,
+                            0.0006595443701371551,
+                            0.000641591555904597,
+                            0.0005168947391211987,
+                            0.0006747929728589952,
+                            0.0006691680755466223,
+                            0.0006565524963662028,
+                            0.000632181647233665,
+                            0.0006619601626880467,
+                            0.000572663324419409,
+                            0.000645424472168088,
+                            0.0006164144724607468,
+                            0.0005629535880871117,
+                            0.0005697975866496563,
+                            0.0006883660098537803,
+                            0.0006199361523613334,
+                            0.0005463651614263654,
+                            0.0006099916063249111,
+                            0.000569057185202837,
+                            0.0005781569052487612,
+                            0.0005226114299148321,
+                            0.0004901915672235191,
+                            0.0005659470334649086,
+                            0.0005618436844088137,
+                            0.0006996651063673198,
+                            0.0005794740281999111,
+                            0.0005557227996177971,
+                            0.0005024878191761672,
+                            0.0006087057990953326,
+                            0.0006718095974065363,
+                            0.0007234778604470193,
+                            0.0006217784830369055,
+                            0.0005217844154685736,
+                            0.0006124366773292422,
+                            0.000663040904328227,
+                            0.0006218517664819956,
+                            0.000564862450119108,
+                            0.0005583419115282595,
+                            0.000574232020881027,
+                            0.0005271151312626898,
+                            0.0005746138631366193,
+                            0.0006236911285668612,
+                            0.0005430203746072948,
+                            0.0005711853737011552,
+                            0.0005796142504550517,
+                            0.0006193070439621806,
+                            0.0005576222320087254,
+                            0.0006076297140680254,
+                            0.0006500107701867819,
+                            0.0006016039405949414,
+                            0.0006265661213546991,
+                            0.0007275307434611022,
+                            0.0006013163947500288,
+                            0.0005914876237511635,
+                            0.0007589313318021595,
+                            0.0005967804463580251,
+                            0.0008645253255963326,
+                            0.0006576986634172499,
+                            0.000565099879167974,
+                            0.0005897182854823768,
+                            0.0005613068933598697,
+                            0.0006575087900273502,
+                            0.0006501955795101821,
+                            0.0007009760593064129,
+                            0.0006192986038513482,
+                            0.0007553540635854006,
+                            0.0006087016081437469,
+                            0.0006187999388203025,
+                            0.0005836684140376747,
+                            0.0006266030250117183,
+                            0.0005712800193578005,
+                            0.0005299962940625846,
+                            0.0005861144745722413,
+                            0.0006012532394379377,
+                            0.0007667723693884909,
+                            0.00048797865747474134,
+                            0.0005843912367708981,
+                            0.000449503306299448,
+                            0.0005257903831079602,
+                            0.0005979117704555392,
+                            0.0005600042059086263,
+                            0.0005547019536606967,
+                            0.0006786355515941978,
+                            0.000570537755265832,
+                            0.0008289284887723625,
+                            0.0006481454474851489,
+                            0.0005616089329123497,
+                            0.0006231772131286561,
+                            0.0006235010805539787,
+                            0.00068769755307585,
+                            0.0005922494456171989,
+                            0.0007631481857970357,
+                            0.0006265131523832679,
+                            0.0005296662566252053,
+                            0.0006817970424890518,
+                            0.0007314616814255714,
+                            0.0005941235576756299,
+                            0.0006151092820800841,
+                            0.0006319874082691967,
+                            0.0006889212527312338,
+                            0.0005953069194220006,
+                            0.0005364923272281885,
+                            0.000548618845641613,
+                            0.0005949722835794091,
+                            0.000540098815690726,
+                            0.0006676797056570649,
+                            0.0006304323906078935,
+                            0.000593740027397871,
+                            0.0005762911168858409,
+                            0.000518303771968931,
+                            0.0005351405125111341,
+                            0.0005899733514524996,
+                            0.0005055050132796168,
+                            0.0004981710808351636,
+                            0.0006912306998856366,
+                            0.0004757080168928951,
+                            0.0005106731550768018,
+                            0.0005662284092977643,
+                            0.000621322076767683,
+                            0.0005540653946809471,
+                            0.0006270595476962626,
+                            0.0005376693443395197,
+                            0.0007237489917315543,
+                            0.0006943327025510371,
+                            0.0005987149779684842,
+                            0.0005542050930671394,
+                            0.0006128338281996548,
+                            0.0005627825739793479,
+                            0.0006101839244365692,
+                            0.0006270374869927764,
+                            0.0005303233629092574,
+                            0.0006418299162760377,
+                            0.0006058605504222214,
+                            0.0005241059698164463,
+                            0.0006144333747215569,
+                            0.0006858668057247996,
+                            0.000545692746527493,
+                            0.0005532580544240773,
+                            0.0005883264821022749,
+                            0.0006319688982330263,
+                            0.0005719088949263096,
+                            0.0005591976223513484,
+                            0.000665154424495995,
+                            0.0007656555972062051,
+                            0.0006008988129906356,
+                            0.0006142068887129426,
+                            0.000696639355737716,
+                            0.0006533333798870444,
+                            0.0006292926846072078,
+                            0.0006590017583221197,
+                            0.0005901032127439976,
+                            0.0007314240210689604,
+                            0.0006031125667504966,
+                            0.0006116436561569571,
+                            0.000646220229100436,
+                            0.0005349426064640284,
+                            0.00069234031252563,
+                            0.0005760613712482154,
+                            0.0006634504534304142,
+                            0.0005666723009198904,
+                            0.0006734668859280646,
+                            0.0006028971984051168,
+                            0.0006857164553366601,
+                            0.0007608385640196502,
+                            0.0004930184804834425,
+                            0.0005146984476596117,
+                            0.0006655369652435184,
+                            0.000676835305057466,
+                            0.0005073903012089431,
+                            0.0005634751287288964,
+                            0.000503869669046253,
+                            0.0005835287156514823,
+                            0.000724300742149353,
+                            0.0005887104198336601,
+                            0.0006561712361872196,
+                            0.0006508297519758344,
+                            0.0006667886045761406,
+                            0.0005354043678380549,
+                            0.0005248135421425104,
+                            0.0005966305616311729,
+                            0.0006311662145890296,
+                            0.0005996398977003992,
+                            0.0005491467891260982,
+                            0.0005595902912318707,
+                            0.0005935707595199347,
+                            0.0005960185662843287,
+                            0.000603392836637795,
+                            0.0006522936164401472,
+                            0.0006435146788135171,
+                            0.0006266941782087088,
+                            0.0005942837451584637,
+                            0.0006371383205987513,
+                            0.0006502108299173415,
+                            0.0006199888302944601,
+                            0.0005101854330860078,
+                            0.000715795555151999,
+                            0.0006697411299683154,
+                            0.0005342445801943541,
+                            0.00046310428297147155,
+                            0.0005586666520684958,
+                            0.0006494020344689488,
+                            0.0006045480258762836,
+                            0.0005842788959853351,
+                            0.0004997635842300951,
+                            0.0007259563426487148,
+                            0.000502281473018229,
+                            0.0006119925528764725,
+                            0.0007478753686882555,
+                            0.0005345342215150595,
+                            0.0006062983302399516,
+                            0.0006094513810239732,
+                            0.0005881984252482653,
+                            0.000563852721825242,
+                            0.000530520745087415,
+                            0.0006339374813251197,
+                            0.0006613848963752389,
+                            0.0006308045121841133,
+                            0.000596549070905894,
+                            0.0006041027372702956,
+                            0.0005226926296018064,
+                            0.0006789453327655792,
+                            0.0006842630682513118,
+                            0.0005330329295247793,
+                            0.0006163918878883123,
+                            0.0005367628764361143,
+                            0.0005430374876596034,
+                            0.000503499642945826,
+                            0.000612557225394994,
+                            0.0007199518731795251,
+                            0.0007224535220302641,
+                            0.0005332300788722932,
+                            0.0005308058462105691,
+                            0.0005020368262194097,
+                            0.0006826416938565671,
+                            0.0005845080595463514,
+                            0.000541362096555531,
+                            0.0005713163409382105,
+                            0.0005828642752021551,
+                            0.0005332852597348392,
+                            0.0005101063870824873,
+                            0.0005480764666572213,
+                            0.000528298260178417,
+                            0.0005022331024520099,
+                            0.0005492630298249424,
+                            0.0005019766394980252,
+                            0.0006367091555148363,
+                            0.0005528312758542597,
+                            0.0007010038243606687,
+                            0.0005010170280002058,
+                            0.000544888200238347,
+                            0.0005169771029613912,
+                            0.0005065334844402969,
+                            0.0006856714026071131,
+                            0.0005776759353466332,
+                            0.0006177772884257138,
+                            0.0005408303113654256,
+                            0.0005913125933147967,
+                            0.0005549562047235668,
+                            0.0006916081765666604,
+                            0.0005551375215873122,
+                            0.0005963758449070156,
+                            0.0006303546251729131,
+                            0.0005785406683571637,
+                            0.0006011610385030508,
+                            0.0005972275394015014,
+                            0.0005574835231527686,
+                            0.0005403548711910844,
+                            0.0005676402361132205,
+                            0.0005350115825422108,
+                            0.0005194729310460389,
+                            0.0007084465469233692,
+                            0.0005996343679726124,
+                            0.0006163682555779815,
+                            0.0005178418941795826,
+                            0.0006029325886629522,
+                            0.0006205763202160597,
+                            0.0006791604100726545,
+                            0.0006605903618037701,
+                            0.0006446724873967469,
+                            0.0005175914266146719,
+                            0.0005926609155721962,
+                            0.0006198149058036506,
+                            0.000638345954939723,
+                            0.0006146647501736879,
+                            0.0006429339991882443,
+                            0.0005617918795906007,
+                            0.0006449546199291945,
+                            0.0005134826060384512,
+                            0.0005301993805915117,
+                            0.0006893367390148342,
+                            0.0005993881495669484,
+                            0.0005503922584466636,
+                            0.0005322702345438302,
+                            0.0006904231267981231,
+                            0.0006325277499854565,
+                            0.0005600574659183621,
+                            0.0005999461282044649,
+                            0.0005501593695953488,
+                            0.0005777559708803892,
+                            0.0006179162301123142,
+                            0.0006284511764533818,
+                            0.0006020618020556867,
+                            0.000594751036260277,
+                            0.0005505997687578201,
+                            0.0005393476458266377,
+                            0.0006731278845109046,
+                            0.000650957110337913,
+                            0.0005771227879449725,
+                            0.0005796792102046311,
+                            0.0005694807623513043,
+                            0.0006300508975982666,
+                            0.0005450787139125168,
+                            0.000506993499584496,
+                            0.0006290358141995966,
+                            0.0007109089056029916,
+                            0.0005274367285892367,
+                            0.00046896212734282017,
+                            0.0005427573341876268,
+                            0.0005785867106169462,
+                            0.0005061686970293522,
+                            0.0005278955795802176,
+                            0.0005880154203623533,
+                            0.0005303936777636409,
+                            0.0005389094003476202,
+                            0.0005209629307501018,
+                            0.0007755393744446337,
+                            0.0006083213957026601,
+                            0.0006326386937871575,
+                            0.0005853472976014018,
+                            0.000539303757250309,
+                            0.00048725341912359,
+                            0.0006274933693930507,
+                            0.0006072031683288515,
+                            0.0005313462461344898,
+                            0.0005837244098074734,
+                            0.0005988542689010501,
+                            0.0006532277329824865,
+                            0.0006130078108981252,
+                            0.0006485973717644811,
+                            0.0005848028813488781,
+                            0.0006061099702492356,
+                            0.0006900681182742119,
+                            0.0006539188325405121,
+                            0.0007266196771524847,
+                            0.0006777531234547496,
+                            0.0006430357461795211,
+                            0.0005470135947689414,
+                            0.000630605558399111,
+                            0.0005270767724141479,
+                            0.0005183535977266729,
+                            0.00067073357058689,
+                            0.000643002858851105,
+                            0.0006823056610301137,
+                            0.0006166023085825145,
+                            0.000605352281127125,
+                            0.0005870703607797623,
+                            0.000626750523224473,
+                            0.0005322804790921509,
+                            0.0006982084014452994,
+                            0.0005972490762360394,
+                            0.0005369968712329865,
+                            0.0006030031945556402,
+                            0.000509655277710408,
+                            0.0005837007774971426,
+                            0.0005653450498357415,
+                            0.0006126348744146526,
+                            0.0005956985405646265,
+                            0.0006025072652846575,
+                            0.0005079533439129591,
+                            0.0005357185727916658,
+                            0.0006312351324595511,
+                            0.0007038371404632926,
+                            0.0005439116503112018,
+                            0.0005056142690591514,
+                            0.0005512023344635963,
+                            0.0005987550830468535,
+                            0.0007844078354537487,
+                            0.0006982171325944364,
+                            0.0005361264920793474,
+                            0.0005964706069789827,
+                            0.0005913224304094911,
+                            0.0006320399115793407,
+                            0.0005861096433363855,
+                            0.0005797466728836298,
+                            0.0006905004265718162,
+                            0.0007078657508827746,
+                            0.0005874750786460936,
+                            0.0005789439310319722,
+                            0.0005956568638794124,
+                            0.000624054460786283,
+                            0.0005351630970835686,
+                            0.0006352681084536016,
+                            0.0005470043397508562,
+                            0.0006420970894396305,
+                            0.0005672352854162455,
+                            0.0006356865633279085,
+                            0.0006291616009548306,
+                            0.0007050313288345933,
+                            0.0005630426458083093,
+                            0.0007162905531004071,
+                            0.0005324266967363656,
+                            0.0007060081698000431,
+                            0.000700214528478682,
+                            0.0006038597202859819,
+                            0.0005382938543334603,
+                            0.0006172754801809788,
+                            0.0005236773868091404,
+                            0.000683587568346411,
+                            0.0005865207058377564,
+                            0.0005373263848014176,
+                            0.0005600166041404009,
+                            0.000492297054734081,
+                            0.000516316678840667,
+                            0.000592787517234683,
+                            0.000607400550507009,
+                            0.0006697652279399335,
+                            0.0005067779566161335,
+                            0.0007485625683329999,
+                            0.0006269037839956582,
+                            0.0006071249954402447,
+                            0.000535095576196909,
+                            0.000625941960606724,
+                            0.0005683811032213271,
+                            0.0005758954212069511,
+                            0.0007537028868682683,
+                            0.0005856459029018879,
+                            0.0005699056200683117,
+                            0.0006720682140439749,
+                            0.0006453705718740821,
+                            0.0005982876755297184,
+                            0.0006401980062946677,
+                            0.000603229331318289,
+                            0.000681973178870976,
+                            0.000513863458763808,
+                            0.0005455354112200439,
+                            0.0005949093028903008,
+                            0.0007119915680959821,
+                            0.0006535337306559086,
+                            0.0005890046595595777,
+                            0.0006533792475238442,
+                            0.0005438061198219657,
+                            0.0005643529584631324,
+                            0.0007152759353630245,
+                            0.000832008954603225,
+                            0.0005976126994937658,
+                            0.0005829644505865872,
+                            0.0005931062623858452,
+                            0.000648838235065341,
+                            0.0005950650665909052,
+                            0.000655107491184026,
+                            0.0005334586603567004,
+                            0.0006242999224923551,
+                            0.0006492161191999912,
+                            0.0005512811476364732,
+                            0.000597979174926877,
+                            0.0006825134623795748,
+                            0.0006320698303170502,
+                            0.0006046456401236355,
+                            0.0005914054927416146,
+                            0.0005141171859577298,
+                            0.0005658225272782147,
+                            0.000674830749630928,
+                            0.00054182019084692,
+                            0.0005822720704600215,
+                            0.0005356142646633089,
+                            0.0006204739329405129,
+                            0.0006844689487479627,
+                            0.0005195550038479269,
+                            0.0006105134962126613,
+                            0.0007020210032351315,
+                            0.0005019963136874139,
+                            0.0004931222065351903,
+                            0.0006466985214501619,
+                            0.0006014114478603005,
+                            0.0005986119504086673,
+                            0.0006772970664314926,
+                            0.0006419667042791843,
+                            0.0005284761427901685,
+                            0.0007162067922763526,
+                            0.0005942295538261533,
+                            0.0006195838213898242,
+                            0.0008533807704225183,
+                            0.0006580635672435164,
+                            0.000628132198471576,
+                            0.0005690568359568715,
+                            0.0007952594314701855,
+                            0.0007587949512526393,
+                            0.0006466896738857031,
+                            0.0006980389007367194,
+                            0.0006707385764457285,
+                            0.0006672192830592394,
+                            0.0005312612047418952,
+                            0.0006954108830541372,
+                            0.000529566896148026,
+                            0.0005698335007764399,
+                            0.0005366691621020436,
+                            0.0005671695689670742,
+                            0.0005300808697938919,
+                            0.0007780473097227514,
+                            0.0005561303696595132,
+                            0.0005681048496626318,
+                            0.0006158014293760061,
+                            0.0006937928847037256,
+                            0.0006372912903316319,
+                            0.0006391264614649117,
+                            0.0005768525879830122,
+                            0.0005110156489536166,
+                            0.0008882717811502516,
+                            0.0006279833614826202,
+                            0.0006433104281313717,
+                            0.0004952612216584384,
+                            0.0007110641454346478,
+                            0.0004763507458847016,
+                            0.0007467904943041503,
+                            0.0005137238185852766,
+                            0.00068334472598508,
+                            0.0005445385468192399,
+                            0.0010081842774525285,
+                            0.0005754859885200858,
+                            0.0005687265656888485,
+                            0.000690060667693615,
+                            0.0009303425322286785,
+                            0.0006822168943472207,
+                            0.0006847240729257464,
+                            0.0006458056741394103,
+                            0.0005849733715876937,
+                            0.0006520699826069176,
+                            0.0006166515522636473,
+                            0.0005908424500375986,
+                            0.0005670478567481041,
+                            0.0006651795702055097,
+                            0.0006596705643460155,
+                            0.0005716781597584486,
+                            0.0007983021205291152,
+                            0.0005677278968505561,
+                            0.0006090239621698856,
+                            0.0006334524368867278,
+                            0.0008164198370650411,
+                            0.0005208729999139905,
+                            0.0005111893406137824,
+                            0.0006941640167497098,
+                            0.0006520276656374335,
+                            0.0005793725722469389,
+                            0.0005262800841592252,
+                            0.0005465596914291382,
+                            0.000592160620726645,
+                            0.0006432526279240847,
+                            0.0007068879785947502,
+                            0.0007167291478253901,
+                            0.0005587887717410922,
+                            0.000543430563993752,
+                            0.000646035885438323,
+                            0.000664999068249017,
+                            0.0007142974063754082,
+                            0.0007595830247737467,
+                            0.0007952132145874202,
+                            0.0007661800482310355,
+                            0.0006799732218496501,
+                            0.0006678784266114235,
+                            0.0006698181969113648,
+                            0.0007591096218675375,
+                            0.0006599864573217928,
+                            0.0006292311591096222,
+                            0.000613853451795876,
+                            0.0007548679714091122,
+                            0.0007409208337776363,
+                            0.0006418815464712679,
+                            0.0005492710624821484,
+                            0.0005717448657378554,
+                            0.0007599546224810183,
+                            0.0006093440460972488,
+                            0.0006393465446308255,
+                            0.0006153128342702985,
+                            0.000830787408631295,
+                            0.0006073190015740693,
+                            0.0007413795101456344,
+                            0.0006467698258347809,
+                            0.0007518032798543572,
+                            0.0006090012029744685,
+                            0.0006607876275666058,
+                            0.0008864067494869232,
+                            0.0005415868363343179,
+                            0.0005641778698191047,
+                            0.0006052333046682179,
+                            0.0006009385688230395,
+                            0.0007650358602404594,
+                            0.0006450059590861201,
+                            0.0006390796042978764,
+                            0.0005879545351490378,
+                            0.0006711087771691382,
+                            0.0005530499620363116,
+                            0.0005938400863669813,
+                            0.0007807956426404417,
+                            0.000648072047624737,
+                            0.0005575857940129936,
+                            0.0006820799317210913,
+                            0.0007376190042123199,
+                            0.000693866575602442,
+                            0.0008147070184350014,
+                            0.0006187161197885871,
+                            0.000620004313532263,
+                            0.0006991724367253482,
+                            0.0005704413633793592,
+                            0.0006664163083769381,
+                            0.0008820194052532315,
+                            0.0006316325161606073,
+                            0.0005837551434524357,
+                            0.000850479758810252,
+                            0.0006891945959068835,
+                            0.0010763260070234537,
+                            0.0006848546909168363,
+                            0.0007178832311183214,
+                            0.0005554465460591018,
+                            0.0005826741689816117,
+                            0.0007759222644381225,
+                            0.0008751201094128191,
+                            0.0006828227778896689,
+                            0.0005794201861135662,
+                            0.0008803882519714534,
+                            0.0008780994103290141,
+                            0.000889540824573487,
+                            0.000545626156963408,
+                            0.000610490096732974,
+                            0.0011343208607286215,
+                            0.0006809305632486939,
+                            0.0006823998992331326,
+                            0.000673270202241838,
+                            0.0008162875892594457,
+                            0.0006713303155265749,
+                            0.0005151330842636526,
+                            0.0008087647147476673,
+                            0.0006401633727364242,
+                            0.0006868012715131044,
+                            0.001337196328677237,
+                            0.0009250573930330575,
+                            0.0005648759542964399,
+                            0.000672156224027276,
+                            0.0007624089485034347,
+                            0.0009861673461273313,
+                            0.000713690125849098,
+                            0.0012214358430355787,
+                            0.000686658953782171,
+                            0.0005438474472612143,
+                            0.0007570366142317653,
+                            0.0006613031146116555,
+                            0.0008720152545720339,
+                            0.0008281295304186642,
+                            0.0004836781299673021,
+                            0.0005996761610731483,
+                            0.0008610773365944624,
+                            0.0007052379078231752,
+                            0.0006872188532724977,
+                            0.0007526365225203335,
+                            0.0007643124554306269,
+                            0.0007620525429956615,
+                            0.0007070312858559191,
+                            0.0005746267270296812,
+                            0.0005216736462898552,
+                            0.000633331248536706,
+                            0.0006274619372561574,
+                            0.0006192631553858519,
+                            0.0006133537972345948,
+                            0.0005596419796347618,
+                            0.0005492414347827435,
+                            0.0005559999262914062,
+                            0.0006773376371711493,
+                            0.0006867506890557706,
+                            0.0009604855440557003,
+                            0.0007645688601769507,
+                            0.0006339604733511806,
+                            0.0005677513545379043,
+                            0.0005676158471032977,
+                            0.0008783322991803288,
+                            0.0005963548901490867,
+                            0.0006623913068324327,
+                            0.0008553714724257588,
+                            0.0007247683242894709,
+                            0.0006454976391978562,
+                            0.0007852634880691767,
+                            0.0007946427213028073,
+                            0.0006975012947805226,
+                            0.0006132784765213728,
+                            0.0007762894383631647,
+                            0.0005751921562477946,
+                            0.0011327053653076291,
+                            0.0007417078595608473,
+                            0.0006080521852709353,
+                            0.0006057584541849792,
+                            0.0009438110864721239,
+                            0.0006353204371407628,
+                            0.0006483488832600415,
+                            0.000568543910048902,
+                            0.0006064654444344342,
+                            0.0006402106373570859,
+                            0.0005944154690951109,
+                            0.0006173080764710903,
+                            0.0007479502237401903,
+                            0.0008162253070622683,
+                            0.0006350248586386442,
+                            0.0005111247883178294,
+                            0.0008381503284908831,
+                            0.0011464882409200072,
+                            0.0006112411501817405,
+                            0.0006565043004229665,
+                            0.0005806780536659062,
+                            0.000737741298507899,
+                            0.0009542358457110822,
+                            0.0006080775638110936,
+                            0.0008674888522364199,
+                            0.0006218694616109133,
+                            0.0006878696731291711,
+                            0.0006373774376697838,
+                            0.0006502497708424926,
+                            0.0006322989356704056,
+                            0.0007634617504663765,
+                            0.0005615366972051561,
+                            0.0007182666449807584,
+                            0.0006561041227541864,
+                            0.000584870926104486,
+                            0.0008502730051986873,
+                            0.0005956939421594143,
+                            0.000610462564509362,
+                            0.0006504464545287192,
+                            0.0005064805736765265,
+                            0.0005991475190967321,
+                            0.0007342839380726218,
+                            0.0006924713379703462,
+                            0.000602451735176146,
+                            0.0006528215599246323,
+                            0.0007203241111710668,
+                            0.000645302701741457,
+                            0.0007338839932344854,
+                            0.0006171326385810971,
+                            0.0006937715806998312,
+                            0.0007121420348994434,
+                            0.0007110977312549949,
+                            0.0006136035080999136,
+                            0.0006174384034238756,
+                            0.0007351015228778124,
+                            0.0006493875989690423,
+                            0.0006281637470237911,
+                            0.0006595923914574087,
+                            0.0008976797689683735,
+                            0.000676581694278866,
+                            0.0006012259400449693,
+                            0.0005197667051106691,
+                            0.0005258352612145245,
+                            0.0005644396878778934,
+                            0.0006815721862949431,
+                            0.0006522223120555282,
+                            0.0006896235863678157,
+                            0.0006809455808252096,
+                            0.0006161394412629306,
+                            0.0006511575193144381,
+                            0.0005735529121011496,
+                            0.000707295082975179,
+                            0.0006225425167940557,
+                            0.0005702116759493947,
+                            0.0009627717081457376,
+                            0.0006280755624175072,
+                            0.0006315757054835558,
+                            0.0005260092439129949,
+                            0.0007224892033264041,
+                            0.001141790417023003,
+                            0.0006296381470747292,
+                            0.0005150394863449037,
+                            0.0005480847903527319,
+                            0.0006912860553711653,
+                            0.0011111587518826127,
+                            0.0006587401148863137,
+                            0.0007893186993896961,
+                            0.0007626426522620022,
+                            0.0007992889732122421,
+                            0.0006575853331014514,
+                            0.0006828086916357279,
+                            0.000735822191927582,
+                            0.000637263001408428,
+                            0.0008495054789818823,
+                            0.000587606627959758,
+                            0.0005963998846709728,
+                            0.0006572831771336496,
+                            0.0005862997495569289,
+                            0.0005590473301708698,
+                            0.000654654751997441,
+                            0.0005347205442376435,
+                            0.000574928242713213,
+                            0.0005289695109240711,
+                            0.0005965505843050778,
+                            0.0006652793963439763,
+                            0.0005946161109022796,
+                            0.0006827845936641097,
+                            0.0008474470814689994,
+                            0.0007960718357935548,
+                            0.0005446374998427927,
+                            0.0006670283619314432,
+                            0.0007817883160896599,
+                            0.0008815684705041349,
+                            0.0005636049900203943,
+                            0.0008405265398323536,
+                            0.0006256182095967233,
+                            0.0005834460025653243,
+                            0.0007650122279301286,
+                            0.0005832062452100217,
+                            0.001431058393791318,
+                            0.0008009634912014008,
+                            0.0005876881768926978,
+                            0.0006299388478510082,
+                            0.0008928259485401213,
+                            0.0005291439592838287,
+                            0.0006265981937758625,
+                            0.000591218180488795,
+                            0.0004952996969223022,
+                            0.0007509653805755079,
+                            0.0007889400003477931,
+                            0.0005861541139893234,
+                            0.0005940087721683085,
+                            0.0005524686421267688,
+                            0.0005829247529618442,
+                            0.0005879347445443273,
+                            0.0007700579008087516,
+                            0.0007019229815341532,
+                            0.0005436757928691804,
+                            0.0005329573759809136,
+                            0.0007167717558331788,
+                            0.0006108724046498537,
+                            0.00104454904794693,
+                            0.0006661208462901413,
+                            0.0005677591543644667,
+                            0.000598762184381485,
+                            0.0008309161639772356,
+                            0.0006765940925106406,
+                            0.0005552878719754517,
+                            0.0005915717920288444,
+                            0.0009432663791812956,
+                            0.0008680448518134654,
+                            0.0006340222898870707,
+                            0.0006241690716706216,
+                            0.0008026946452446282,
+                            0.000738123431801796,
+                            0.0007037769537419081,
+                            0.0005835797637701035,
+                            0.0005607056082226336,
+                            0.0009591965936124325,
+                            0.0004977283533662558,
+                            0.0007098226342350245,
+                            0.001019938150420785,
+                            0.0006207139231264591,
+                            0.0007178749656304717,
+                            0.000983917503617704,
+                            0.000620224978774786,
+                            0.0006041127489879727,
+                            0.000533362734131515,
+                            0.0005887657171115279,
+                            0.0006382555002346635,
+                            0.0008329444099217653,
+                            0.0006473010289482772,
+                            0.0009714791085571051,
+                            0.000638331868685782,
+                            0.0004977242206223309,
+                            0.0009534484706819057,
+                            0.0005618216237053275,
+                            0.0008205071208067238,
+                            0.0006631349097006023,
+                            0.0005583615275099874,
+                            0.000572937831748277,
+                            0.0005736198509112,
+                            0.0006249109283089638,
+                            0.0006382755236700177,
+                            0.0008648454677313566,
+                            0.0006950513343326747,
+                            0.0005431175814010203,
+                            0.0005124841118231416,
+                            0.0006359739345498383,
+                            0.0006920368759892881,
+                            0.0005864885170012712,
+                            0.0007928866543807089,
+                            0.0005575079121626914,
+                            0.0007287449552677572,
+                            0.000720528420060873,
+                            0.0005654124543070793,
+                            0.0005773153388872743,
+                            0.0005434551858343184,
+                            0.0006908729556016624,
+                            0.0006168935215100646,
+                            0.000607831752859056,
+                            0.0006801012787036598,
+                            0.0006186649552546442,
+                            0.0005683141062036157,
+                            0.0008884874987415969,
+                            0.0006489110528491437,
+                            0.0006291275494731963,
+                            0.0006213620654307306,
+                            0.0005711414851248264,
+                            0.0006322781555354595,
+                            0.0006849882192909718,
+                            0.000642720959149301,
+                            0.0006581355119124055,
+                            0.0005937662208452821,
+                            0.0005901015247218311,
+                            0.0006953434785827994,
+                            0.0006644125678576529,
+                            0.0005818017525598407,
+                            0.0009316573268733919,
+                            0.0005057375528849661,
+                            0.0005262367194518447,
+                            0.0007991456659510732,
+                            0.000636218348518014,
+                            0.0005659383605234325,
+                            0.000606374756898731,
+                            0.0006677297642454505,
+                            0.0006330699543468654,
+                            0.000548870419152081,
+                            0.0005957866669632494,
+                            0.0008154063252732158,
+                            0.0006082358304411173,
+                            0.0005588626954704523,
+                            0.0006818118272349238,
+                            0.0005310559645295143,
+                            0.000676603289321065,
+                            0.0005579734570346773,
+                            0.0006186465034261346,
+                            0.0005896257935091853,
+                            0.0005708217504434288,
+                            0.0005848417058587074,
+                            0.0006431422079913318,
+                            0.0006403058650903404,
+                            0.0007582313846796751,
+                            0.00053335604025051,
+                            0.0007022514473646879,
+                            0.0005753979203291237,
+                            0.0008007962605915964,
+                            0.0007808397058397532,
+                            0.0006454868125729263,
+                            0.0007088634301908314,
+                            0.0007116633933037519,
+                            0.0006086652865633368,
+                            0.0006496316636912525,
+                            0.0004930275026708841,
+                            0.0006561814225278795,
+                            0.0006397970719262958,
+                            0.0007308434578590095,
+                            0.0007519262144342065,
+                            0.0007314714603126049,
+                            0.0005690283142030239,
+                            0.0005595320835709572,
+                            0.0005943759460933506,
+                            0.0006760117830708623,
+                            0.000628672307357192,
+                            0.0006079362938180566,
+                            0.0007333686808124185,
+                            0.0006474568508565426,
+                            0.0006574873114004731,
+                            0.0006335244979709387,
+                            0.0006406051688827574,
+                            0.0007034403970465064,
+                            0.0005561048747040331,
+                            0.0009693672182038426,
+                            0.000578927923925221,
+                            0.000669590022880584,
+                            0.000673431612085551,
+                            0.0005936712259426713,
+                            0.0007426074589602649,
+                            0.0006497359718196094,
+                            0.0005823704414069653,
+                            0.0006976915756240487,
+                            0.0006497455760836601,
+                            0.0006459834985435009,
+                            0.0005411768797785044,
+                            0.0007611101609654725,
+                            0.0007132269092835486,
+                            0.0006167321116663516,
+                            0.0005893249181099236,
+                            0.000689683307427913,
+                            0.0006434132810682058,
+                            0.000592690019402653,
+                            0.0005796939949505031,
+                            0.0005514859221875668
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 61,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "2": {
+                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type_is_module": false,
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 62,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.03270483389496803
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.03068052977323532
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 63,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.03605125471949577
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "3": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.V_linears.1",
+                "input_tensor_infos": [
+                    {
+                        "id": 63,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.03605125471949577
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.001063563278876245,
+                            0.000840360764414072,
+                            0.001179258688353002,
+                            0.0009192316792905331,
+                            0.0013406308135017753,
+                            0.001240442623384297,
+                            0.0008043090929277241,
+                            0.0008632134413346648,
+                            0.0012126716319471598,
+                            0.0010784914484247565,
+                            0.0008845540578477085,
+                            0.0010434744181111455,
+                            0.001050072954967618,
+                            0.0009848627960309386,
+                            0.001013482571579516,
+                            0.0010965639958158135,
+                            0.0007752657984383404,
+                            0.0012314190389588475,
+                            0.0008214119588956237,
+                            0.0010175565257668495,
+                            0.00121410028077662,
+                            0.000982180587016046,
+                            0.0008273079874925315,
+                            0.0011967449681833386,
+                            0.0008586324984207749,
+                            0.0011927811428904533,
+                            0.0009316467912867665,
+                            0.0007285660831257701,
+                            0.0009530813549645245,
+                            0.00098419317509979,
+                            0.0010687365429475904,
+                            0.0008447769214399159,
+                            0.0009009987115859985,
+                            0.001043836586177349,
+                            0.0009949320228770375,
+                            0.0011076944647356868,
+                            0.0011951811611652374,
+                            0.0008661698666401207,
+                            0.0010229289764538407,
+                            0.0015628013061359525,
+                            0.0009140724432654679,
+                            0.0011039597447961569,
+                            0.0007973593892529607,
+                            0.0010302797891199589,
+                            0.0007468352559953928,
+                            0.0012698801001533866,
+                            0.0009026427287608385,
+                            0.0009831564966589212,
+                            0.0009091774700209498,
+                            0.0007749104406684637,
+                            0.0007961082155816257,
+                            0.0008862227550707757,
+                            0.0009896806441247463,
+                            0.0010791739914566278,
+                            0.0012782602570950985,
+                            0.0010745107429102063,
+                            0.0007474828162230551,
+                            0.0007400984177365899,
+                            0.0011179089779034257,
+                            0.0014006824931129813,
+                            0.0008097535464912653,
+                            0.0008035232895053923,
+                            0.0007849325775168836,
+                            0.0009733984479680657,
+                            0.0007272318471223116,
+                            0.001125178299844265,
+                            0.0008933257195167243,
+                            0.0007931109284982085,
+                            0.0010553766041994095,
+                            0.0010222229175269604,
+                            0.0010733137605711818,
+                            0.0010303976014256477,
+                            0.0007816388970240951,
+                            0.0012798390816897154,
+                            0.0009291816386394203,
+                            0.001174796256236732,
+                            0.0008727587410248816,
+                            0.0008293803548440337,
+                            0.0010067758848890662,
+                            0.000852470810059458,
+                            0.0009519770974293351,
+                            0.0009429209167137742,
+                            0.0009828414767980576,
+                            0.0011496638180688024,
+                            0.0013468008255586028,
+                            0.000992610934190452,
+                            0.0013378505827859044,
+                            0.0013028508983552456,
+                            0.0009443956660106778,
+                            0.0011895339703187346,
+                            0.0010047547984868288,
+                            0.001037313137203455,
+                            0.0013059880584478378,
+                            0.0014507423620671034,
+                            0.0009686646517366171,
+                            0.0008209935622289777,
+                            0.0010667790193110704,
+                            0.0013591530732810497,
+                            0.0012798154493793845,
+                            0.0011328132823109627,
+                            0.0012983011547476053,
+                            0.0012121024774387479,
+                            0.0008791995933279395,
+                            0.0013152892934158444,
+                            0.0010754131944850087,
+                            0.0012878517154604197,
+                            0.0013846216024830937,
+                            0.0010422643972560763,
+                            0.0006420305580832064,
+                            0.0010250058257952332,
+                            0.001173927797935903,
+                            0.0009437931585125625,
+                            0.0009228967246599495,
+                            0.0008892002515494823,
+                            0.0007460970664396882,
+                            0.0010791782988235354,
+                            0.001081739435903728,
+                            0.0009471105295233428,
+                            0.000937595556024462,
+                            0.0011658795410767198,
+                            0.0008096544770523906,
+                            0.0009185124654322863,
+                            0.0009825320448726416,
+                            0.0008367057889699936,
+                            0.0011043485719710588,
+                            0.0010745155159384012,
+                            0.0008668038644827902,
+                            0.0009588583488948643,
+                            0.0010658762184903026,
+                            0.0014484311686828732,
+                            0.001197206205688417,
+                            0.001207103137858212,
+                            0.0007857835735194385,
+                            0.001505261636339128,
+                            0.0011023490224033594,
+                            0.0007566642016172409,
+                            0.0009190604323521256,
+                            0.0009418263216502964,
+                            0.0011388232233002782,
+                            0.0010010498808696866,
+                            0.0007690630736760795,
+                            0.0010931374272331595,
+                            0.0009502629400230944,
+                            0.0010801764437928796,
+                            0.0008034233469516039,
+                            0.0011653497349470854,
+                            0.0006712457397952676,
+                            0.0008731019333936274,
+                            0.0008173389360308647,
+                            0.0008064321591518819,
+                            0.0010995828779414296,
+                            0.0009010481298901141,
+                            0.0010341707384213805,
+                            0.0007848968962207437,
+                            0.0013881834456697106,
+                            0.0013228468596935272,
+                            0.0010563291143625975,
+                            0.0008622646564617753,
+                            0.001361977425403893,
+                            0.001267675426788628,
+                            0.0011056263465434313,
+                            0.000822737580165267,
+                            0.0009111445397138596,
+                            0.0013117502676323056,
+                            0.0011510404292494059,
+                            0.0008375201723538339,
+                            0.0008643710752949119,
+                            0.0007222354179248214,
+                            0.0009038384887389839,
+                            0.000986549654044211,
+                            0.0009356134105473757,
+                            0.0007527319830842316,
+                            0.001529203844256699,
+                            0.0011303271166980267,
+                            0.0009291018941439688,
+                            0.0011341264471411705,
+                            0.0010048924013972282,
+                            0.0007911032298579812,
+                            0.0008656743448227644,
+                            0.0008619035943411291,
+                            0.0008319714688695967,
+                            0.0008445623680017889,
+                            0.000759014452341944,
+                            0.001035219756886363,
+                            0.0011246786452829838,
+                            0.0011450545862317085,
+                            0.001085040275938809,
+                            0.0009479171130806208,
+                            0.0012291165767237544,
+                            0.0009379548719152808,
+                            0.001077949651516974,
+                            0.0017768124816939235,
+                            0.0009081142488867044,
+                            0.0007384842028841376,
+                            0.0009065200574696064,
+                            0.0006671955343335867,
+                            0.0009848743211477995,
+                            0.0010196794755756855,
+                            0.000812670448794961,
+                            0.000972716195974499,
+                            0.0007331420783884823,
+                            0.0007787612266838551,
+                            0.0008381615625694394,
+                            0.0007784173358231783,
+                            0.0007192260236479342,
+                            0.0011137683177366853,
+                            0.001068480429239571,
+                            0.0008429262088611722,
+                            0.0013974327594041824,
+                            0.0008494008798152208,
+                            0.0011288108071312308,
+                            0.0010837846202775836,
+                            0.0011702501215040684,
+                            0.0009576529846526682,
+                            0.0007392297848127782,
+                            0.0010297865374013782,
+                            0.001005487865768373,
+                            0.0010776068083941936,
+                            0.0010519345523789525,
+                            0.00088985834736377,
+                            0.0009974086424335837,
+                            0.0006653285818174481,
+                            0.0012874791864305735,
+                            0.0010638105450198054,
+                            0.000952675414737314,
+                            0.0013194727944210172,
+                            0.0010676963720470667,
+                            0.0006992873968556523,
+                            0.0007238615071401,
+                            0.001014194916933775,
+                            0.0010841151233762503,
+                            0.0009820815175771713,
+                            0.0011136156972497702,
+                            0.0013843851629644632,
+                            0.0009306403226219118,
+                            0.0010991538874804974,
+                            0.0007321216398850083,
+                            0.0008774485904723406,
+                            0.0011579536367207766,
+                            0.001253244816325605,
+                            0.0010341143934056163,
+                            0.0008905697031877935,
+                            0.0009419593843631446,
+                            0.0009697708883322775,
+                            0.0007238167454488575,
+                            0.0012053865939378738,
+                            0.0011445562122389674,
+                            0.001020286581479013,
+                            0.0013683756114915013,
+                            0.0010668092872947454,
+                            0.001102764392271638,
+                            0.0009366601589135826,
+                            0.001404819544404745,
+                            0.0012016353430226445,
+                            0.0011322476202622056,
+                            0.0007652578642591834,
+                            0.0008619757136330009,
+                            0.0009896098636090755,
+                            0.0012275745393708348,
+                            0.0009514171397313476,
+                            0.0010582436807453632,
+                            0.0009328059968538582,
+                            0.0009664004319347441,
+                            0.0009213544544763863,
+                            0.0008216499118134379,
+                            0.001085525145754218,
+                            0.001236760406754911,
+                            0.0011157316621392965,
+                            0.0008354318561032414,
+                            0.0009091042447835207,
+                            0.0009673204622231424,
+                            0.000846229842863977,
+                            0.0008229638333432376,
+                            0.0008828812860883772,
+                            0.001014277688227594,
+                            0.0009310872410424054,
+                            0.001102329115383327,
+                            0.0009102112962864339,
+                            0.0011076421942561865,
+                            0.0009950695093721151,
+                            0.0009481465094722807,
+                            0.0008687367080710828,
+                            0.0008615109254606068,
+                            0.0010390871902927756,
+                            0.0010135245975106955,
+                            0.0007412958075292408,
+                            0.001002752804197371,
+                            0.0012722002575173974,
+                            0.0008017899817787111,
+                            0.0009667389676906168,
+                            0.0012768554734066129,
+                            0.001038596616126597,
+                            0.0013072340516373515,
+                            0.000701194629073143,
+                            0.000892655283678323,
+                            0.0010278510162606835,
+                            0.0012318392982706428,
+                            0.0014706318033859134,
+                            0.0008426929707638919,
+                            0.0010227484162896872,
+                            0.0008694459102116525,
+                            0.001291485968977213,
+                            0.0010358913568779826,
+                            0.0013458169996738434,
+                            0.0011726898374035954,
+                            0.0008537768735550344,
+                            0.0009095689747482538,
+                            0.0008460115059278905,
+                            0.0011271688854321837,
+                            0.0007487339898943901,
+                            0.0010073308367282152,
+                            0.0006306840805336833,
+                            0.0008529736660420895,
+                            0.0008221145835705101,
+                            0.000993944238871336,
+                            0.0008918936946429312,
+                            0.001083896728232503,
+                            0.0010090860305354,
+                            0.0012591950362548232,
+                            0.0009805428562685847,
+                            0.0012068788055330515,
+                            0.000797534070443362,
+                            0.0009729472803883255,
+                            0.0011341884965077043,
+                            0.001723392168059945,
+                            0.0010459052864462137,
+                            0.0008818620699457824,
+                            0.0009220338542945683,
+                            0.0012665202375501394,
+                            0.0010251300409436226,
+                            0.001042851246893406,
+                            0.0010700569255277514,
+                            0.0010213555069640279,
+                            0.000933800358325243,
+                            0.0009517946746200323,
+                            0.0008636226411908865,
+                            0.0008841665694490075,
+                            0.0009631607681512833,
+                            0.0012854152591899037,
+                            0.0009643911034800112,
+                            0.0009619643096812069,
+                            0.001215914380736649,
+                            0.0011813509045168757,
+                            0.0009990654652938247,
+                            0.0008898811065591872,
+                            0.0012197209289297462,
+                            0.0014411512529477477,
+                            0.0008430631714873016,
+                            0.001286245882511139,
+                            0.001044945209287107,
+                            0.0010084961540997028,
+                            0.0010624530259519815,
+                            0.0009362525888718665,
+                            0.0014222109457477927,
+                            0.0010768614010885358,
+                            0.0012135215802118182,
+                            0.0012111712712794542,
+                            0.0010889465920627117,
+                            0.000945541774854064,
+                            0.000883694039657712,
+                            0.0008136354153975844,
+                            0.0011566074099391699,
+                            0.001197867444716394,
+                            0.0008624748443253338,
+                            0.0010729777859523892,
+                            0.0009985248325392604,
+                            0.0008420938975177705,
+                            0.0010314028477296233,
+                            0.0009496417478658259,
+                            0.0012786907609552145,
+                            0.0008180205477401614,
+                            0.0009989677928388119,
+                            0.0010266617173328996,
+                            0.0012008032063022256,
+                            0.0010326900519430637,
+                            0.001002951175905764,
+                            0.0007337394636124372,
+                            0.0008111746283248067,
+                            0.0008449020097032189,
+                            0.0008748172549530864,
+                            0.0011770090786740184,
+                            0.0012188763357698917,
+                            0.0007977476343512535,
+                            0.0010610573226585984,
+                            0.001172437216155231,
+                            0.0008856151252985001,
+                            0.0013650550972670317,
+                            0.0011790958233177662,
+                            0.0009248384740203619,
+                            0.0011975298402830958,
+                            0.0012058322317898273,
+                            0.0008076050435192883,
+                            0.0009051054366864264,
+                            0.0011323989601805806,
+                            0.0012779254466295242,
+                            0.0009427633485756814,
+                            0.0010925326496362686,
+                            0.0010419575264677405,
+                            0.0014879335649311543,
+                            0.0009185575763694942,
+                            0.0010008245008066297,
+                            0.0008321345085278153,
+                            0.0010205385042354465,
+                            0.0009612347348593175,
+                            0.0010244438890367746,
+                            0.0010488502448424697,
+                            0.0009554150747135282,
+                            0.0007656820234842598,
+                            0.0007009837427176535,
+                            0.0007309322827495635,
+                            0.0015420299023389816,
+                            0.0008091561612673104,
+                            0.0010815055575221777,
+                            0.0009394146618433297,
+                            0.0008970160852186382,
+                            0.0007592337788082659,
+                            0.0009599156328476965,
+                            0.0012842314317822456,
+                            0.0011279497994109988,
+                            0.0009152001002803445,
+                            0.00116811937186867,
+                            0.0010333553655073047,
+                            0.000887997099198401,
+                            0.0006943396874703467,
+                            0.0010823738994076848,
+                            0.0010368546936661005,
+                            0.001426464761607349,
+                            0.0009133483981713653,
+                            0.001349551253952086,
+                            0.0008248618105426431,
+                            0.0010271079372614622,
+                            0.000924524269066751,
+                            0.0009114181739278138,
+                            0.001169013325124979,
+                            0.0008566135074943304,
+                            0.0010847350349649787,
+                            0.0011054022470489144,
+                            0.0011293531861156225,
+                            0.0011357999173924327,
+                            0.0009492436074651778,
+                            0.0011530871270224452,
+                            0.0009352454799227417,
+                            0.0011500336695462465,
+                            0.0011980206472799182,
+                            0.0008034319034777582,
+                            0.001200519734993577,
+                            0.000893502205144614,
+                            0.0009087168145924807,
+                            0.0011991935316473246,
+                            0.0010079528437927365,
+                            0.0016100391512736678,
+                            0.0009089714731089771,
+                            0.000979712582193315,
+                            0.0008590483921580017,
+                            0.0008649536175653338,
+                            0.0008083882858045399,
+                            0.001059280359186232,
+                            0.0011525849113240838,
+                            0.0008241596515290439,
+                            0.0009335700888186693,
+                            0.000923663261346519,
+                            0.0010982166277244687,
+                            0.0010697513353079557,
+                            0.0008149831555783749,
+                            0.0008842485258355737,
+                            0.000790094374679029,
+                            0.0009564956417307258,
+                            0.0008796608890406787,
+                            0.0008589006029069424,
+                            0.0009449380449950695,
+                            0.0009616404422558844,
+                            0.0009916599374264479,
+                            0.0010388987138867378,
+                            0.0010804124176502228,
+                            0.0008323289803229272,
+                            0.001017836038954556,
+                            0.0007430187542922795,
+                            0.0008891547913663089,
+                            0.00132178352214396,
+                            0.0011266747023910284,
+                            0.0012647861149162054,
+                            0.0010414901189506054,
+                            0.0009256969788111746,
+                            0.0008066743612289429,
+                            0.0009648751583881676,
+                            0.0012099526356905699,
+                            0.0008058748790062964,
+                            0.0012310738675296307,
+                            0.0010008251992985606,
+                            0.0008025525021366775,
+                            0.0010543366661295295,
+                            0.0010203836718574166,
+                            0.0012071910314261913,
+                            0.0009350148611702025,
+                            0.0007760602165944874,
+                            0.0011208392679691315,
+                            0.0012283045798540115,
+                            0.0011916235089302063,
+                            0.0008273086277768016,
+                            0.0008982276194728911,
+                            0.0007734132232144475,
+                            0.0012297576759010553,
+                            0.0008153090020641685,
+                            0.0009868169436231256,
+                            0.0010560972150415182,
+                            0.0011034886119887233,
+                            0.0012077841674908996,
+                            0.0008067796006798744,
+                            0.0009214875171892345,
+                            0.0010580974631011486,
+                            0.000947754830121994,
+                            0.0008845392731018364
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 64,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.01538123469799757
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "4": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.W_linears.1",
+                "input_tensor_infos": [
+                    {
+                        "id": 64,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.01538123469799757
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0010001382324844599,
+                            0.0008801991934888065,
+                            0.0007694984669797122,
+                            0.0010217668022960424,
+                            0.0009664959507063031,
+                            0.0006636077305302024,
+                            0.0007388405501842499,
+                            0.0005566245527006686,
+                            0.0007175784558057785,
+                            0.0013734388630837202,
+                            0.0005870824679732323,
+                            0.0008062038104981184,
+                            0.0007255465025082231,
+                            0.000616938981693238,
+                            0.0005161020671948791,
+                            0.0005474074860103428,
+                            0.0005718526663258672,
+                            0.000935273477807641,
+                            0.0012210996355861425,
+                            0.0007101327064447105,
+                            0.0005479743122123182,
+                            0.0005997379776090384,
+                            0.0005281772464513779,
+                            0.0005083468277007341,
+                            0.0006115892319940031,
+                            0.0006214893655851483,
+                            0.0007549296133220196,
+                            0.0005412492319010198,
+                            0.0006453210371546447,
+                            0.0004928302369080484,
+                            0.0005705422372557223,
+                            0.0008758229669183493,
+                            0.0005363592645153403,
+                            0.000655506388284266,
+                            0.0006337681552395225,
+                            0.0006362447165884078,
+                            0.0006160257034935057,
+                            0.0005608006031252444,
+                            0.000512806698679924,
+                            0.0006982079357840121,
+                            0.0009294716874137521,
+                            0.00110178894829005,
+                            0.0009299525991082191,
+                            0.000653338385745883,
+                            0.0008185419719666243,
+                            0.0012803145218640566,
+                            0.0005835731863044202,
+                            0.0005221095052547753,
+                            0.0008555486565455794,
+                            0.0013887768145650625,
+                            0.0007572761969640851,
+                            0.0005700977053493261,
+                            0.0006763629498891532,
+                            0.0009912870591506362,
+                            0.0005475172656588256,
+                            0.0006650552968494594,
+                            0.0005485909059643745,
+                            0.0005343927768990397,
+                            0.0012860981514677405,
+                            0.0005236768047325313,
+                            0.000595633580815047,
+                            0.0010880626505240798,
+                            0.0005920346011407673,
+                            0.0005865081329829991,
+                            0.001071872073225677,
+                            0.0005800578510388732,
+                            0.0006071650423109531,
+                            0.00129258306697011,
+                            0.0005562282167375088,
+                            0.0005714491126127541,
+                            0.0005568520864471793,
+                            0.0007898195181041956,
+                            0.00107985595241189,
+                            0.0006013807142153382,
+                            0.0011287849629297853,
+                            0.0006798738031648099,
+                            0.0005804119864478707,
+                            0.0005273752030916512,
+                            0.0004942703526467085,
+                            0.000553836754988879,
+                            0.0006346300360746682,
+                            0.0006557197775691748,
+                            0.0005193112883716822,
+                            0.0006799449911341071,
+                            0.00067422742722556,
+                            0.0006269786972552538,
+                            0.0006094748969189823,
+                            0.0011917181545868516,
+                            0.0010519709903746843,
+                            0.0007342339376918972,
+                            0.0006339383544400334,
+                            0.0006347001763060689,
+                            0.0005282337078824639,
+                            0.0006542277405969799,
+                            0.0009781070984899998,
+                            0.0004826173826586455,
+                            0.0010810798266902566,
+                            0.0006260559312067926,
+                            0.0005431104800663888,
+                            0.0014352761209011078,
+                            0.0007328441133722663,
+                            0.0009228643612004817,
+                            0.0006661891820840538,
+                            0.0005771627766080201,
+                            0.000584557477850467,
+                            0.0008988804183900356,
+                            0.0005482317064888775,
+                            0.0005007252912037075,
+                            0.0011924720602110028,
+                            0.0005796134355477989,
+                            0.0008016069186851382,
+                            0.0004985227133147418,
+                            0.0008350407588295639,
+                            0.0005407374701462686,
+                            0.0006983441417105496,
+                            0.0005171856610104442,
+                            0.0005480523686856031,
+                            0.0009131280239671469,
+                            0.0005856317584402859,
+                            0.0011523127323016524,
+                            0.00067959661828354,
+                            0.0005592959932982922,
+                            0.0006112763076089323,
+                            0.0004678087425418198,
+                            0.0006196046015247703,
+                            0.0005405841511674225,
+                            0.000596517464146018,
+                            0.0005613685352727771,
+                            0.0007132022292353213,
+                            0.0006965465727262199,
+                            0.0006957523291930556,
+                            0.0006013281526975334,
+                            0.0006370444316416979,
+                            0.0006735100178048015,
+                            0.0005567849730141461,
+                            0.0006225297693163157,
+                            0.0006292603793554008,
+                            0.00073749985313043,
+                            0.0006136968149803579,
+                            0.0008608735515736043,
+                            0.000593503937125206,
+                            0.0007072854205034673,
+                            0.0007075481116771698,
+                            0.0005716252489946783,
+                            0.0007506898837164044,
+                            0.000725712045095861,
+                            0.0006426053587347269,
+                            0.0006448180065490305,
+                            0.0006273018079809844,
+                            0.0005650551174767315,
+                            0.0006139192846603692,
+                            0.0006901040324009955,
+                            0.0005938748363405466,
+                            0.0006200366769917309,
+                            0.0007783691980876029,
+                            0.0006367158493958414,
+                            0.0006465078913606703,
+                            0.0005016957293264568,
+                            0.0005931808846071362,
+                            0.0005690398975275457,
+                            0.0006182685610838234,
+                            0.000588876660913229,
+                            0.0005179769359529018,
+                            0.0006797584937885404,
+                            0.0006186763639561832,
+                            0.0006038637948222458,
+                            0.0005912412889301777,
+                            0.000645842868834734,
+                            0.0007963958196341991,
+                            0.0006245268741622567,
+                            0.0008431170717813075,
+                            0.0005760645726695657,
+                            0.00071177096106112,
+                            0.0010724698659032583,
+                            0.0004948198329657316,
+                            0.0007792958640493453,
+                            0.0006579512264579535,
+                            0.0009405531454831362,
+                            0.0006076984573155642,
+                            0.0008799100760370493,
+                            0.0005599710857495666,
+                            0.000544241804163903,
+                            0.0007096866029314697,
+                            0.0006154615548439324,
+                            0.0008423354011029005,
+                            0.000646343338303268,
+                            0.0005469833849929273,
+                            0.0008007084252312779,
+                            0.0006024825852364302,
+                            0.000713190995156765,
+                            0.0006729293963871896,
+                            0.0008457544026896358,
+                            0.0006957361474633217,
+                            0.0005429155426099896,
+                            0.000892645213752985,
+                            0.0006646147812716663,
+                            0.0006844419986009598,
+                            0.0005853099282830954,
+                            0.0005931998020969331,
+                            0.0007597277872264385,
+                            0.0006548939854837954,
+                            0.0007104473188519478,
+                            0.0005652769468724728,
+                            0.0007439841865561903,
+                            0.0005436045466922224,
+                            0.0006277122301980853,
+                            0.0006547615048475564,
+                            0.0007972971652634442,
+                            0.0007893024594523013,
+                            0.0007601210963912308,
+                            0.0006789945182390511,
+                            0.0006943964981473982,
+                            0.0007347629871219397,
+                            0.0006162566714920104,
+                            0.0006919922889210284,
+                            0.0007886854000389576,
+                            0.0005608024657703936,
+                            0.0005346917896531522,
+                            0.0006281137466430664,
+                            0.0006528383819386363,
+                            0.0006496438873000443,
+                            0.0005754100275225937,
+                            0.0008425083942711353,
+                            0.0007483690860681236,
+                            0.000642509083263576,
+                            0.0005888416199013591,
+                            0.0005661399918608367,
+                            0.000675778544973582,
+                            0.0008349150884896517,
+                            0.0005811862065456808,
+                            0.0006273301551118493,
+                            0.0008349692216143012,
+                            0.0007781089516356587,
+                            0.00088182941544801,
+                            0.000789993500802666,
+                            0.0006563416100107133,
+                            0.0005347583210095763,
+                            0.0007554600015282631,
+                            0.0006833029328845441,
+                            0.0008699740283191204,
+                            0.0006040561129339039,
+                            0.000658024859149009,
+                            0.0005594771937467158,
+                            0.000652546645142138,
+                            0.0007554886979050934,
+                            0.0006842711591161788,
+                            0.0006329328170977533,
+                            0.0006122574559412897,
+                            0.0006088698864914477,
+                            0.0007241967832669616,
+                            0.0007382537587545812,
+                            0.0009154459112323821,
+                            0.0006046264315955341,
+                            0.0005895156064070761,
+                            0.0006136061274446547,
+                            0.0005824328982271254,
+                            0.0007494339370168746,
+                            0.0006193030276335776,
+                            0.0006150850676931441,
+                            0.0008557654218748212,
+                            0.0007177512743510306,
+                            0.0007258654804900289,
+                            0.0006114591960795224,
+                            0.0005976086249575019,
+                            0.0008134124800562859,
+                            0.0006390261114574969,
+                            0.0006332910270430148,
+                            0.0007939574425108731,
+                            0.0009228167473338544,
+                            0.0006798107642680407,
+                            0.0007404109928756952,
+                            0.0006692394381389022,
+                            0.0007466920069418848,
+                            0.0009448691853322089,
+                            0.0008343609515577555,
+                            0.0007376907742582262,
+                            0.0006741699180565774,
+                            0.0005732748541049659,
+                            0.0006886951741762459,
+                            0.0006326263537630439,
+                            0.0011415898334234953,
+                            0.0007093214080668986,
+                            0.0010020238114520907,
+                            0.0006834375089965761,
+                            0.0006378692924045026,
+                            0.0006875076796859503,
+                            0.0006772347842343152,
+                            0.0007488943520002067,
+                            0.0006683908286504447,
+                            0.0005263775237835944,
+                            0.00100261764600873,
+                            0.00101341656409204,
+                            0.0005702917114831507,
+                            0.0005437898798845708,
+                            0.0007988036959432065,
+                            0.0006095373537391424,
+                            0.0007298743003048003,
+                            0.0008802676456980407,
+                            0.0006062797037884593,
+                            0.0009847097098827362,
+                            0.0007083896198309958,
+                            0.0006569554097950459,
+                            0.0009424681775271893,
+                            0.0006663106032647192,
+                            0.0005848479922860861,
+                            0.0006694397307001054,
+                            0.0008205585763789713,
+                            0.000566614733543247,
+                            0.0008238634327426553,
+                            0.0010077684419229627,
+                            0.00085520523134619,
+                            0.0006866552284918725,
+                            0.0007047988474369049,
+                            0.0006253402098082006,
+                            0.0007523039821535349,
+                            0.0008371495641767979,
+                            0.0007722901646047831,
+                            0.0008070343756116927,
+                            0.0009443463059142232,
+                            0.0006390766357071698,
+                            0.0007408828823827207,
+                            0.0006253799074329436,
+                            0.0006915387348271906,
+                            0.000753816042561084,
+                            0.0007302044541575015,
+                            0.0006808537873439491,
+                            0.0006019361899234354,
+                            0.0010251314379274845,
+                            0.0006975164287723601,
+                            0.0007744381437078118,
+                            0.0007763393223285675,
+                            0.0005522036226466298,
+                            0.0005292596761137247,
+                            0.0005669321399182081,
+                            0.00114249880425632,
+                            0.0008994920644909143,
+                            0.0009273610776290298,
+                            0.0007684954325668514,
+                            0.0006437338888645172,
+                            0.0007244224543683231,
+                            0.0006885171169415116,
+                            0.0008489638566970825,
+                            0.0011386628029868007,
+                            0.0008181330631487072,
+                            0.0007553233299404383,
+                            0.001201212638989091,
+                            0.0009213802986778319,
+                            0.0007489852141588926,
+                            0.0006100566824898124,
+                            0.0005984794697724283,
+                            0.00058463477762416,
+                            0.0007894032169133425,
+                            0.0008170164655894041,
+                            0.0007127668941393495,
+                            0.0004939074860885739,
+                            0.0005821482045575976,
+                            0.0010322938906028867,
+                            0.0012164503568783402,
+                            0.0005439117667265236,
+                            0.0009899786673486233,
+                            0.0006716272328048944,
+                            0.0008314885199069977,
+                            0.0006229876889847219,
+                            0.0005920620751567185,
+                            0.0007412197883240879,
+                            0.0007220502011477947,
+                            0.0006343686254695058,
+                            0.0005990045610815287,
+                            0.0009706561104394495,
+                            0.001049639075063169,
+                            0.0008097165264189243,
+                            0.0006344944704324007,
+                            0.0006363522261381149,
+                            0.0006054789992049336,
+                            0.0007549344445578754,
+                            0.0005876472569070756,
+                            0.0008326898096129298,
+                            0.0008946220623329282,
+                            0.0005744753871113062,
+                            0.0007316754781641066,
+                            0.0007898305193521082,
+                            0.000724178971722722,
+                            0.0009390098275616765,
+                            0.0006026309565640986,
+                            0.0013103531673550606,
+                            0.0011542066931724548,
+                            0.0008573512313887477,
+                            0.0007316144765354693,
+                            0.0008240995812229812,
+                            0.0006872491212561727,
+                            0.000591983029153198,
+                            0.0007348363869823515,
+                            0.0010699925478547812,
+                            0.0007465392700396478,
+                            0.0009050689986906946,
+                            0.0006501267198473215,
+                            0.001022831303998828,
+                            0.0007514097378589213,
+                            0.0008490305626764894,
+                            0.0017096736701205373,
+                            0.001023144111968577,
+                            0.0006008580676279962,
+                            0.000927664979826659,
+                            0.001014070468954742,
+                            0.0006075134733691812,
+                            0.0009816838428378105,
+                            0.000941102160140872,
+                            0.0007360014715231955,
+                            0.001049020909704268,
+                            0.0007830066024325788,
+                            0.001164053799584508,
+                            0.0009689541766420007,
+                            0.0009716956410557032,
+                            0.0007674043881706893,
+                            0.0012831886997446418,
+                            0.0008568134508095682,
+                            0.0011007154826074839,
+                            0.001111188204959035,
+                            0.0010256614768877625,
+                            0.0008468841551803052,
+                            0.0006054372061043978,
+                            0.0008159524877555668,
+                            0.0010967678390443325,
+                            0.000996889197267592,
+                            0.0006142597994767129,
+                            0.0010602767579257488,
+                            0.0008079007966443896,
+                            0.0009978520683944225,
+                            0.0006818044348619878,
+                            0.0008224552148021758,
+                            0.0007517198100686073,
+                            0.0009315719362348318,
+                            0.0011703063501045108,
+                            0.0005419671069830656,
+                            0.00081173877697438,
+                            0.0010432320414111018,
+                            0.0009595572482794523,
+                            0.0010005318326875567,
+                            0.00065251300111413,
+                            0.0006408314220607281,
+                            0.0007647883030585945,
+                            0.0010717123514041305,
+                            0.000857186911161989,
+                            0.0008842161041684449,
+                            0.000785197364166379,
+                            0.0009775847429409623,
+                            0.000984171056188643,
+                            0.0010216694790869951,
+                            0.0006865783943794668,
+                            0.0007246041786856949,
+                            0.0006861158180981874,
+                            0.0008779173367656767,
+                            0.000741634692531079,
+                            0.0008469551685266197,
+                            0.00069201341830194,
+                            0.0005529361660592258,
+                            0.0005589885404333472,
+                            0.0006534193526022136,
+                            0.0007124794065020978,
+                            0.0012740783859044313,
+                            0.0011516211088746786,
+                            0.0011527459137141705,
+                            0.0009570755646564066,
+                            0.0009812094504013658,
+                            0.000747038924600929,
+                            0.0009007721091620624,
+                            0.0010949772549793124,
+                            0.0010716929100453854,
+                            0.0009588223765604198,
+                            0.001174770062789321,
+                            0.0015765350544825196,
+                            0.0007822136394679546,
+                            0.0007553335162810981,
+                            0.0008785175159573555,
+                            0.000752550782635808,
+                            0.0009919148869812489,
+                            0.0010389693779870868,
+                            0.0006027651252225041,
+                            0.0007963695679791272,
+                            0.0009876427939161658,
+                            0.0005797615158371627,
+                            0.0005486660520546138,
+                            0.0007504824316129088,
+                            0.0008399103535339236,
+                            0.000555006496142596,
+                            0.0007084187236614525,
+                            0.0006241206428967416,
+                            0.0009085616329684854,
+                            0.0007181033142842352,
+                            0.0007218869868665934,
+                            0.0006131274276413023,
+                            0.0006196049507707357,
+                            0.000642571016214788,
+                            0.0007196137448772788,
+                            0.0005901886033825576,
+                            0.001241380232386291,
+                            0.0006580078625120223,
+                            0.0007252530194818974,
+                            0.001023565884679556,
+                            0.0010103300446644425,
+                            0.0008897075895220041,
+                            0.0007362391916103661,
+                            0.0007362474570982158,
+                            0.0008193663088604808,
+                            0.0007046418613754213,
+                            0.0009003621526062489,
+                            0.0011655150447040796,
+                            0.0010448534740135074,
+                            0.0006818597903475165,
+                            0.0008004431147128344,
+                            0.001056797569617629,
+                            0.0009701710077933967,
+                            0.0006016867118887603,
+                            0.0005004271515645087,
+                            0.000612421368714422,
+                            0.0005522481515072286,
+                            0.0005103605799376965,
+                            0.0007839727913960814,
+                            0.0005088606267236173,
+                            0.0006164689548313618,
+                            0.0005746501265093684,
+                            0.0006810665945522487,
+                            0.0005372529267333448,
+                            0.0005733089055866003,
+                            0.0005665072822012007,
+                            0.0005914861685596406,
+                            0.0006026493501849473,
+                            0.0005568331689573824,
+                            0.0005134145030751824,
+                            0.0005817989585921168,
+                            0.0005947498138993979,
+                            0.000577972736209631,
+                            0.0006163246580399573,
+                            0.000588279333896935,
+                            0.0005020907265134156,
+                            0.0005475046345964074,
+                            0.0006119284080341458,
+                            0.0005373034509830177,
+                            0.0005595055408775806,
+                            0.0005236006691120565,
+                            0.0006023106398060918,
+                            0.0005463113193400204,
+                            0.0005165844922885299,
+                            0.0005755676538683474,
+                            0.0006337257218547165,
+                            0.0005200508167035878,
+                            0.0006527970544993877,
+                            0.0006267608841881156,
+                            0.0006169463158585131,
+                            0.0006016790866851807,
+                            0.0006409644847735763,
+                            0.0006833462975919247,
+                            0.0006465394981205463,
+                            0.0005714173312298954,
+                            0.0005757897743023932,
+                            0.0005180523730814457,
+                            0.0006477997521869838,
+                            0.0006115360883995891,
+                            0.0006366982124745846,
+                            0.0006698442157357931,
+                            0.0005595936672762036,
+                            0.0005402519600465894,
+                            0.0005748591502197087,
+                            0.0005126058240421116,
+                            0.0005995415849611163,
+                            0.0005242577171884477,
+                            0.000585053872782737,
+                            0.0005152566591277719,
+                            0.0005362721858546138,
+                            0.0008513223147019744,
+                            0.0005451691104099154,
+                            0.0005514504737220705,
+                            0.0006346341106109321,
+                            0.0005615477566607296,
+                            0.0006005740724503994,
+                            0.0005931828054599464,
+                            0.0005487842718139291,
+                            0.0005234145210124552,
+                            0.00058673822786659,
+                            0.0006249527796171606,
+                            0.0005829092697240412,
+                            0.0005788661073893309,
+                            0.0005960490088909864,
+                            0.0005632330430671573,
+                            0.0005848604487255216,
+                            0.0005490718758665025,
+                            0.0005528597394004464,
+                            0.0005466315778903663,
+                            0.0006709714070893824,
+                            0.0005431097815744579,
+                            0.000494494684971869,
+                            0.0005484004504978657,
+                            0.0006839740672148764,
+                            0.0006856544059701264,
+                            0.0005225237109698355,
+                            0.0005715213483199477,
+                            0.0005769944982603192,
+                            0.0006637285696342587,
+                            0.0005196498241275549,
+                            0.00048217541188932955,
+                            0.0006515116547234356,
+                            0.0005379655049182475,
+                            0.000635695003438741,
+                            0.0006324133137241006,
+                            0.0006374951917678118,
+                            0.0007564017432741821,
+                            0.000574951118323952,
+                            0.0005444424459710717,
+                            0.0005177374696359038,
+                            0.0005772529984824359,
+                            0.0007061798241920769,
+                            0.0006614492158405483,
+                            0.0006431132205761969,
+                            0.0006015258841216564,
+                            0.0006470293737947941,
+                            0.0005387514247559011,
+                            0.0006980561302043498,
+                            0.0005541120190173388,
+                            0.0006275982013903558,
+                            0.0005779957282356918,
+                            0.0005642065079882741,
+                            0.0007165236165747046,
+                            0.0005708016105927527,
+                            0.0005987549084238708,
+                            0.0006407637847587466,
+                            0.0008106525638140738,
+                            0.0005601843586191535,
+                            0.0005913383793085814,
+                            0.0005701091722585261,
+                            0.0005763256340287626,
+                            0.0005589722422882915,
+                            0.0005716001614928246,
+                            0.0005378374480642378,
+                            0.0005742241628468037,
+                            0.0005906371516175568,
+                            0.0006513920379802585,
+                            0.0006745690479874611,
+                            0.0006338092498481274,
+                            0.000528897566255182,
+                            0.0007635602378286421,
+                            0.0005945844459347427,
+                            0.0006703208782710135,
+                            0.0006626222748309374,
+                            0.0005750621785409749,
+                            0.000605589768383652,
+                            0.0006005163304507732,
+                            0.0005633860128000379,
+                            0.0005383634124882519,
+                            0.0006892183446325362,
+                            0.0006489016814157367,
+                            0.0006891994271427393,
+                            0.0005526092718355358,
+                            0.0006521422765217721,
+                            0.0007288216729648411,
+                            0.0005596881965175271,
+                            0.0005244138883426785,
+                            0.0005627320497296751,
+                            0.0005486733862198889,
+                            0.0007358831353485584,
+                            0.0006854664534330368,
+                            0.0005187903298065066,
+                            0.0006226202240213752,
+                            0.0005621207528747618,
+                            0.0007272141519933939,
+                            0.0005252424161881208,
+                            0.0006832515937276185,
+                            0.0006165393278934062,
+                            0.0006824189913459122,
+                            0.000574206467717886,
+                            0.0006200497737154365,
+                            0.0005223688785918057,
+                            0.0005237233708612621,
+                            0.000649983761832118,
+                            0.000677077507134527,
+                            0.0006289908778853714,
+                            0.0005220357561483979,
+                            0.000570974254515022,
+                            0.0005712045240215957,
+                            0.0005533518851734698,
+                            0.00064815953373909,
+                            0.0006523093907162547,
+                            0.0007262274157255888,
+                            0.0006800920236855745,
+                            0.0005394036998040974,
+                            0.000611984811257571,
+                            0.0005526416352950037,
+                            0.0005779039929620922,
+                            0.0005017660441808403,
+                            0.0005182873574085534,
+                            0.0006395140080712736,
+                            0.0006663186941295862,
+                            0.0007520069484598935,
+                            0.00044264752068556845,
+                            0.000603365886490792,
+                            0.0005548094632104039,
+                            0.0006086011417210102,
+                            0.0006851458456367254,
+                            0.0005166054470464587,
+                            0.0005778600461781025,
+                            0.0006188047700561583,
+                            0.0004926942056044936,
+                            0.0006330792675726116,
+                            0.0004912480362690985,
+                            0.0005398994544520974,
+                            0.0005010999739170074,
+                            0.0005446625291369855,
+                            0.0005404384573921561,
+                            0.0006069444352760911,
+                            0.0005535503732971847,
+                            0.0005434611230157316,
+                            0.0004859502660110593,
+                            0.000687283871229738,
+                            0.0005966793978586793,
+                            0.0006015372928231955,
+                            0.000602862739469856,
+                            0.0005905291181989014,
+                            0.0006455773254856467,
+                            0.0005255049327388406,
+                            0.0006377785466611385,
+                            0.0006305109709501266,
+                            0.0005630503292195499,
+                            0.0006278838845901191,
+                            0.0005482432898133993,
+                            0.0005744824302382767,
+                            0.0005473983474075794,
+                            0.000629162066616118,
+                            0.000519755354616791,
+                            0.0005445348797366023,
+                            0.0006098075537011027,
+                            0.0005869335727766156,
+                            0.0005953313666395843,
+                            0.0004828068194910884,
+                            0.0005787322297692299,
+                            0.0006473764660768211,
+                            0.0006404488813132048,
+                            0.0004925191751681268,
+                            0.0005248442175798118,
+                            0.0005551158683374524,
+                            0.0005136404652148485,
+                            0.0006624067900702357,
+                            0.0006437597912736237,
+                            0.0007066014222800732,
+                            0.0005035222275182605,
+                            0.0006656189798377454,
+                            0.0005900816177017987,
+                            0.0005891623441129923,
+                            0.0005520741688087583,
+                            0.0006475646514445543,
+                            0.000497909146361053,
+                            0.0005940924747847021,
+                            0.0006502857431769371,
+                            0.0005484677967615426,
+                            0.0005732874851673841,
+                            0.0005612471140921116,
+                            0.000578991137444973,
+                            0.0006205064710229635,
+                            0.000587975315283984,
+                            0.0007121528615243733,
+                            0.0006760212127119303,
+                            0.0006424197927117348,
+                            0.000743525626603514,
+                            0.0006906699272803962,
+                            0.0005953862564638257,
+                            0.0004971559392288327,
+                            0.0006380063132382929,
+                            0.0006998272729106247,
+                            0.0007019518525339663,
+                            0.0006884442991577089,
+                            0.0005854727933183312,
+                            0.0005050201434642076,
+                            0.0005632693646475673,
+                            0.00056844373466447,
+                            0.0005611292435787618,
+                            0.0005536905955523252,
+                            0.0005795039469376206,
+                            0.0006220308132469654,
+                            0.0005605886690318584,
+                            0.0005486903828568757,
+                            0.0005344192031770945,
+                            0.0005592136294580996,
+                            0.0006309032905846834,
+                            0.0005214754492044449,
+                            0.0005917306989431381,
+                            0.0005997131229378283,
+                            0.00059482705546543,
+                            0.0006461036391556263,
+                            0.00048118471750058234,
+                            0.0006346689187921584,
+                            0.0005287271342240274,
+                            0.0005786156980320811,
+                            0.0006069426308386028,
+                            0.0005945719894953072,
+                            0.0005223019397817552,
+                            0.0005721508641727269,
+                            0.0006391346687451005,
+                            0.0005995973479002714,
+                            0.0005989295314066112,
+                            0.0006368919857777655,
+                            0.0005792457377538085,
+                            0.0005923167918808758,
+                            0.0008478923118673265,
+                            0.0006292401230894029,
+                            0.000597493490204215,
+                            0.0005730139091610909,
+                            0.0006241698283702135,
+                            0.0006697300705127418,
+                            0.0006985366344451904,
+                            0.0006134248687885702,
+                            0.0006324979476630688,
+                            0.0005667596124112606,
+                            0.0005326287355273962,
+                            0.0005869636661373079,
+                            0.0005179245490580797,
+                            0.0004859009641222656,
+                            0.0008270003600046039,
+                            0.0006356471567414701,
+                            0.0006104062777012587,
+                            0.0006229451391845942,
+                            0.0007412114064209163,
+                            0.0006450958317145705,
+                            0.0006841690046712756,
+                            0.0006625158712267876,
+                            0.000574823294300586,
+                            0.0005525585729628801,
+                            0.0005729305557906628,
+                            0.0006003656308166683,
+                            0.0006337303202599287,
+                            0.0006452338420785964,
+                            0.0006369085167534649,
+                            0.0006316222134046257,
+                            0.0005528710316866636,
+                            0.0005409059813246131,
+                            0.000589788134675473,
+                            0.0005356573383323848,
+                            0.0006014619139023125,
+                            0.000611410359852016,
+                            0.0006297258660197258,
+                            0.0005965702584944665,
+                            0.0004980870871804655,
+                            0.0006057324353605509,
+                            0.0006803395808674395,
+                            0.0006135172443464398,
+                            0.0005868570297025144,
+                            0.0006816145032644272,
+                            0.0005079936236143112,
+                            0.0006816372042521834,
+                            0.0006381355924531817,
+                            0.000539477332495153,
+                            0.000538172316737473,
+                            0.000541188579518348,
+                            0.0006180427153594792,
+                            0.000703808618709445,
+                            0.0005187211208976805,
+                            0.0006594365113414824,
+                            0.0005997714470140636,
+                            0.0006271071615628898,
+                            0.0005924868164584041,
+                            0.0006471693050116301,
+                            0.0005404383991844952,
+                            0.0006259591900743544,
+                            0.0005490475450642407,
+                            0.0006585590308532119,
+                            0.000612808158621192,
+                            0.0006303113186731935,
+                            0.0007148852455429733,
+                            0.0005961268325336277,
+                            0.000866963411681354,
+                            0.0005188578506931663,
+                            0.0005867266445420682,
+                            0.0006915460107848048,
+                            0.0005896263173781335,
+                            0.0005181803135201335,
+                            0.0007157206418924034,
+                            0.0005983104347251356,
+                            0.0007004398503340781,
+                            0.0006071811076253653,
+                            0.0005654694396071136,
+                            0.0005668742815032601,
+                            0.0005200327723287046,
+                            0.0005207429639995098,
+                            0.0006388099864125252,
+                            0.0005215444252826273,
+                            0.0005244330386631191,
+                            0.0005913906497880816,
+                            0.0005572811351157725,
+                            0.0006949164089746773,
+                            0.0006042505847290158,
+                            0.0005305468221195042,
+                            0.000564811285585165,
+                            0.0005686177173629403,
+                            0.0005330460844561458,
+                            0.0005944973090663552,
+                            0.000540645036380738,
+                            0.000581068976316601,
+                            0.0006993751158006489,
+                            0.0005249506793916225,
+                            0.0009153041755780578,
+                            0.0005606979830190539,
+                            0.0010027397656813264,
+                            0.0008981297141872346,
+                            0.0005582633893936872,
+                            0.0007695002132095397,
+                            0.0008024300914257765,
+                            0.0005993579397909343,
+                            0.0008507359307259321,
+                            0.00057452660985291,
+                            0.0007836787845008075,
+                            0.0013568820431828499,
+                            0.0009511894313618541,
+                            0.0009669137070886791,
+                            0.0005931621417403221,
+                            0.0009064209880307317,
+                            0.0006135134608484805,
+                            0.0008250706596300006,
+                            0.0006976453587412834,
+                            0.000991165405139327,
+                            0.0007479711202904582,
+                            0.0008535572560504079,
+                            0.000901492137927562,
+                            0.0008557421970181167,
+                            0.0008030782919377089,
+                            0.0007891522254794836,
+                            0.0011221821187064052,
+                            0.0007683855947107077,
+                            0.0012302183313295245,
+                            0.0007666901801712811,
+                            0.0007600957760587335,
+                            0.0009672933374531567,
+                            0.0006854892708361149,
+                            0.0009399253176525235,
+                            0.0006952381227165461,
+                            0.0008788544218987226,
+                            0.0009888848289847374,
+                            0.0005966495955362916,
+                            0.000919709331355989,
+                            0.0012491862289607525,
+                            0.0010290095815435052,
+                            0.0006698865327052772,
+                            0.0009374465444125235,
+                            0.0008981255232356489,
+                            0.0007217153324745595,
+                            0.0007842374034225941,
+                            0.0008081634296104312,
+                            0.0009150752448476851,
+                            0.0006025231559760869,
+                            0.0009751911275088787,
+                            0.0007208618917502463,
+                            0.0008444863487966359,
+                            0.0009637923212721944,
+                            0.0007853772840462625,
+                            0.0010076183825731277,
+                            0.0008955553639680147,
+                            0.0007539301295764744,
+                            0.0006460310542024672,
+                            0.0009873691014945507,
+                            0.0011775229359045625,
+                            0.0006460213917307556,
+                            0.0006035035476088524,
+                            0.0008771534194238484,
+                            0.0005939010879956186,
+                            0.0007472590077668428,
+                            0.0011800858192145824,
+                            0.0009102183976210654,
+                            0.0008438215591013432,
+                            0.0007533218595199287,
+                            0.0008423363324254751,
+                            0.0012504764599725604,
+                            0.0006381725543178618,
+                            0.0007810468086972833,
+                            0.000674727198202163,
+                            0.0006489974330179393,
+                            0.000985937425866723,
+                            0.0006067960057407618,
+                            0.0006808214238844812,
+                            0.0006310777389444411,
+                            0.0009170514531433582,
+                            0.0006568669923581183,
+                            0.0009806867456063628,
+                            0.0011014165356755257,
+                            0.0005038395174778998,
+                            0.000613824580796063,
+                            0.0008761546923778951,
+                            0.0008038119412958622,
+                            0.0007382080657407641,
+                            0.0008014090708456933,
+                            0.0006555135478265584,
+                            0.0010360751766711473,
+                            0.0011262571206316352,
+                            0.0007490362622775137,
+                            0.0007084982353262603,
+                            0.0008099229889921844,
+                            0.000647500972263515,
+                            0.0008208091603592038,
+                            0.0005763684748671949,
+                            0.0005256821168586612,
+                            0.001237406861037016,
+                            0.0005008350126445293,
+                            0.0007400150643661618,
+                            0.0005751272547058761,
+                            0.0005961451679468155,
+                            0.0007280738791450858,
+                            0.0008617062703706324,
+                            0.0007548964349552989,
+                            0.0005935559165664017,
+                            0.0009801872074604034,
+                            0.0007808181107975543,
+                            0.0008479789830744267,
+                            0.0006934732664376497,
+                            0.0013117798371240497,
+                            0.0007711555226705968,
+                            0.0009164619259536266,
+                            0.0007614570786245167,
+                            0.000524889852385968,
+                            0.0011624505277723074,
+                            0.0006713424809277058,
+                            0.0008405271219089627,
+                            0.00097640510648489,
+                            0.0006309321615844965,
+                            0.0005266565131023526,
+                            0.0013915584422647953,
+                            0.0010301800211891532,
+                            0.0006107004592195153,
+                            0.0006809385376982391,
+                            0.0006517341244034469,
+                            0.0008116111275739968,
+                            0.0006240718648768961,
+                            0.0005057105445303023,
+                            0.0006398708210326731,
+                            0.0006733490736223757,
+                            0.0007273563533090055,
+                            0.0005683209747076035,
+                            0.0005723533104173839,
+                            0.0005653376574628055,
+                            0.0006525068311020732,
+                            0.0006085397326387465,
+                            0.0005989756900817156,
+                            0.0008893051999621093,
+                            0.0006652940646745265,
+                            0.0008261665352620184,
+                            0.000700031581800431,
+                            0.0008126049651764333,
+                            0.0005928577738814056,
+                            0.0005466574220918119,
+                            0.0006800285773351789,
+                            0.0007462783833034337,
+                            0.0007033100118860602,
+                            0.0007554586045444012,
+                            0.0006377407116815448,
+                            0.0006212302250787616,
+                            0.0006657253252342343,
+                            0.0006492113461717963,
+                            0.0007255017990246415,
+                            0.0008822198142297566,
+                            0.0006225415272638202,
+                            0.0005166816990822554,
+                            0.0005940970149822533,
+                            0.0007538784993812442,
+                            0.0007387161022052169,
+                            0.0005597915733233094,
+                            0.0009005229803733528,
+                            0.0008477236842736602,
+                            0.0005213601980358362,
+                            0.0006177047616802156,
+                            0.000671774207148701,
+                            0.0006168215768411756,
+                            0.0006530754035338759,
+                            0.0006862804875709116,
+                            0.0006274818442761898,
+                            0.0005864654667675495,
+                            0.0005844901315867901,
+                            0.0006023234454914927,
+                            0.000591448275372386,
+                            0.0006382925203070045,
+                            0.0005741546046920121,
+                            0.000582985463552177,
+                            0.0007170753669925034,
+                            0.0007112746243365109,
+                            0.0007559976656921208,
+                            0.0007886865641921759,
+                            0.0006546364747919142,
+                            0.0006570269470103085,
+                            0.0006827438483014703,
+                            0.0006254457985050976,
+                            0.0007256846292875707,
+                            0.0008862234535627067,
+                            0.0007528303540311754,
+                            0.0005815215408802032,
+                            0.00049042102182284,
+                            0.0009459397988393903,
+                            0.0008509336039423943,
+                            0.0005353087326511741,
+                            0.00081005843821913,
+                            0.0007948366110213101,
+                            0.0008025210117921233,
+                            0.0007468895055353642,
+                            0.0005068244645372033,
+                            0.0005911775515414774,
+                            0.000941177480854094,
+                            0.0005330120329745114,
+                            0.0006370997289195657,
+                            0.0009597872267477214,
+                            0.0007722494774498045,
+                            0.0009721059468574822,
+                            0.0006839476991444826,
+                            0.00063824572134763,
+                            0.0007952836458571255,
+                            0.00057883036788553,
+                            0.0007267419714480639,
+                            0.000713156652636826,
+                            0.000592285708989948,
+                            0.0008320003398694098,
+                            0.0007576217758469284,
+                            0.0006138184107840061,
+                            0.0005491357296705246,
+                            0.0005372249870561063,
+                            0.0008735537412576377,
+                            0.0006028675707057118,
+                            0.0007751245284453034,
+                            0.0006579404580406845,
+                            0.0007525304099544883,
+                            0.0005350526189431548,
+                            0.0005572994123212993,
+                            0.0006563269998878241,
+                            0.0006549402605742216,
+                            0.0008406810229644179,
+                            0.0005769107374362648,
+                            0.0006738203228451312,
+                            0.0007183988345786929,
+                            0.0008130233618430793,
+                            0.0006106895743869245,
+                            0.0006437400006689131,
+                            0.0008743333164602518,
+                            0.000638601602986455,
+                            0.0005533341900445521,
+                            0.0005978142144158483,
+                            0.0006105388165451586,
+                            0.0005765277310274541,
+                            0.0006118935998529196,
+                            0.0007406722870655358,
+                            0.0007617220981046557,
+                            0.0005889880703762174,
+                            0.0005907726008445024,
+                            0.0006559890462085605,
+                            0.0006557219894602895,
+                            0.0006890525110065937,
+                            0.0006185261881910264,
+                            0.0007165379356592894,
+                            0.0007950295694172382,
+                            0.0005482806009240448,
+                            0.0006474892725236714,
+                            0.0006473122048191726,
+                            0.0007171550532802939,
+                            0.000608502421528101,
+                            0.0006270296289585531,
+                            0.000632143986877054,
+                            0.0005612341337837279,
+                            0.000605231209192425,
+                            0.00048666828661225736,
+                            0.0005691435653716326,
+                            0.0005877653602510691,
+                            0.0006824976298958063,
+                            0.0006620464264415205,
+                            0.000839324202388525,
+                            0.0005949464393779635,
+                            0.000857136445119977,
+                            0.0006502409232780337,
+                            0.0007770570809952915,
+                            0.0010715319076552987,
+                            0.0006276883068494499,
+                            0.000925680564250797,
+                            0.0006810037884861231,
+                            0.000583619112148881,
+                            0.0008199827861972153,
+                            0.0007051320280879736,
+                            0.0006994014838710427,
+                            0.0005441355751827359,
+                            0.0005189659423194826,
+                            0.0005895630456507206,
+                            0.0006796377128921449,
+                            0.0007004685467109084,
+                            0.000547005794942379,
+                            0.000892903539352119,
+                            0.0006946234498172998,
+                            0.0006073172553442419,
+                            0.00057798478519544,
+                            0.0006725626299157739,
+                            0.0006144510698504746,
+                            0.0007091764127835631,
+                            0.0005389335565268993,
+                            0.0006737691583111882,
+                            0.0007086670957505703,
+                            0.0006427767220884562,
+                            0.0006176293245516717,
+                            0.0007209420436993241,
+                            0.0005723784561268985,
+                            0.0005877965013496578,
+                            0.0006109264795668423,
+                            0.0006306472350843251,
+                            0.0006938958540558815,
+                            0.0005796918412670493,
+                            0.0006680970545858145,
+                            0.0005758918123319745,
+                            0.0005787930567748845,
+                            0.0006095836288295686,
+                            0.0006093480042181909,
+                            0.0007366810459643602,
+                            0.0007209376199170947,
+                            0.000864555942825973,
+                            0.0005495727527886629,
+                            0.0007605147548019886,
+                            0.0007082785596139729,
+                            0.0006125556537881494,
+                            0.0005093703512102365,
+                            0.0006048385985195637,
+                            0.000632865761872381,
+                            0.0005202123429626226,
+                            0.0007014464354142547,
+                            0.0005738167674280703,
+                            0.0006482699536718428,
+                            0.0006531558465212584,
+                            0.0006400699494406581,
+                            0.0005762696382589638,
+                            0.0006438030395656824,
+                            0.0005677919834852219,
+                            0.0006257399800233543,
+                            0.0006356721860356629,
+                            0.0005377607885748148,
+                            0.000556432525627315,
+                            0.0005223425687290728,
+                            0.0009764903225004673,
+                            0.0005906664882786572,
+                            0.0007371290121227503,
+                            0.0006254130857996643,
+                            0.000600453931838274,
+                            0.0006017531850375235,
+                            0.0006674665492027998,
+                            0.0007380217430181801,
+                            0.0005393876344896853,
+                            0.0005440383683890104,
+                            0.0005696219741366804,
+                            0.0006013239617459476,
+                            0.0008945072768256068,
+                            0.000857586448546499,
+                            0.0006684738909825683,
+                            0.0005425839917734265,
+                            0.0006556711159646511,
+                            0.0006886221235617995,
+                            0.0006186450482346117,
+                            0.0006997978198342025,
+                            0.0006508550141006708,
+                            0.0005782541120424867,
+                            0.0006359508261084557,
+                            0.0005684462958015501,
+                            0.0004902370274066925,
+                            0.0005504970322363079,
+                            0.0005565378232859075,
+                            0.0007157257641665637,
+                            0.0006010658107697964,
+                            0.0005570239736698568,
+                            0.0005752341821789742,
+                            0.00062898313626647,
+                            0.0006075030541978776,
+                            0.0005389624275267124,
+                            0.0005464216810651124,
+                            0.0005775540485046804,
+                            0.0006164937512949109,
+                            0.0008301298366859555,
+                            0.0006793990614823997,
+                            0.0006530621903948486,
+                            0.0006591366836801171,
+                            0.0008100158884190023,
+                            0.000589085859246552,
+                            0.0007079575443640351,
+                            0.0005644027260132134,
+                            0.0005723178619518876,
+                            0.0006617282633669674,
+                            0.0007350450614467263,
+                            0.0007003741338849068,
+                            0.0005833070608787239,
+                            0.0007079274510033429,
+                            0.0006551796104758978,
+                            0.0006147524691186845,
+                            0.0005353674641810358,
+                            0.00059150205925107,
+                            0.0005395081243477762,
+                            0.0006949197850190103,
+                            0.0006629210547544062,
+                            0.0005602770252153277,
+                            0.0006037350976839662,
+                            0.0006303454283624887,
+                            0.0007080170325934887,
+                            0.0005855801282450557,
+                            0.0006278955261223018,
+                            0.0007220971747301519,
+                            0.0007160290260799229,
+                            0.0005980517016723752,
+                            0.000676528608892113,
+                            0.0005671472754329443,
+                            0.000601620995439589,
+                            0.0006422232836484909,
+                            0.000654111267067492,
+                            0.0008334445301443338,
+                            0.000634858210105449,
+                            0.0006580572226084769,
+                            0.0006750539760105312,
+                            0.0006394429365172982,
+                            0.00058843003353104,
+                            0.0006467864150181413,
+                            0.0006320232641883194,
+                            0.0008029993041418493,
+                            0.000504770374391228,
+                            0.0005955409724265337,
+                            0.000633468443993479,
+                            0.0006041213637217879,
+                            0.0005689321551471949,
+                            0.0005697945016436279,
+                            0.0008006520802155137,
+                            0.0005421363166533411,
+                            0.0006804127478972077,
+                            0.0006572550046257675,
+                            0.000838003063108772,
+                            0.0006163857760839164,
+                            0.0005950300837866962,
+                            0.0005417062784545124,
+                            0.0007319739670492709,
+                            0.000529979239217937,
+                            0.0007467400282621384,
+                            0.000756902969442308,
+                            0.000601933104917407,
+                            0.00051780795911327,
+                            0.0005880158860236406,
+                            0.0005605478654615581,
+                            0.0005663857446052134,
+                            0.0006643693195655942,
+                            0.0006636266480199993,
+                            0.0007376016583293676,
+                            0.0006345673464238644,
+                            0.0005390068399719894,
+                            0.000575912359636277,
+                            0.0005325418896973133,
+                            0.0005760001367889345,
+                            0.0006988410605117679,
+                            0.0006882268353365362,
+                            0.0005519597907550633,
+                            0.0006150404806248844,
+                            0.000628373003564775,
+                            0.0005400563823059201,
+                            0.0006103807827457786,
+                            0.0005468925228342414,
+                            0.0007017520838417113,
+                            0.0006591032724827528,
+                            0.0006276369094848633,
+                            0.0007689724443480372,
+                            0.0005589441279880702,
+                            0.0006957479054108262,
+                            0.000639227859210223,
+                            0.000686035375110805,
+                            0.0007125702104531229,
+                            0.000596079567912966,
+                            0.0005357854533940554,
+                            0.0005805266555398703,
+                            0.0006525437929667532,
+                            0.0007139951921999454,
+                            0.0005501671694219112,
+                            0.0005791435833089054,
+                            0.0005835581687279046,
+                            0.0006554303108714521,
+                            0.0007764396141283214,
+                            0.0006037952261976898,
+                            0.00071111781289801,
+                            0.0006128053646534681,
+                            0.0006702320533804595,
+                            0.0005728387623094022,
+                            0.0006115207215771079,
+                            0.0008229686063714325,
+                            0.0005535348900593817,
+                            0.000734678644221276,
+                            0.0005775484605692327,
+                            0.0006018573767505586,
+                            0.0006703779217787087,
+                            0.0006670235889032483,
+                            0.0005677926819771528,
+                            0.0005985904135741293,
+                            0.000547882285900414,
+                            0.0006003670860081911,
+                            0.0005542932194657624,
+                            0.0006622575456276536,
+                            0.000529447803273797,
+                            0.0006505209021270275,
+                            0.0007580332458019257,
+                            0.0006686368142254651,
+                            0.0007052354048937559,
+                            0.000870225892867893,
+                            0.0008527408353984356,
+                            0.00056841952027753,
+                            0.0005413693725131452,
+                            0.0005717899766750634,
+                            0.0006664685788564384,
+                            0.0005129416240379214,
+                            0.0005615248810499907,
+                            0.0004855190054513514,
+                            0.0006016766419634223,
+                            0.0008106495952233672,
+                            0.0005437073414213955,
+                            0.0006807101890444756,
+                            0.0005240599275566638,
+                            0.0007019431213848293,
+                            0.0006632315926253796,
+                            0.0007306271581910551,
+                            0.0006274573388509452,
+                            0.0007955132168717682,
+                            0.0005907597369514406,
+                            0.0005668297526426613,
+                            0.0008529055048711598,
+                            0.0007510465220548213,
+                            0.0005852581816725433,
+                            0.0008340900531038642,
+                            0.0005839595105499029,
+                            0.0007922907243482769,
+                            0.0006373666110448539,
+                            0.0008410537848249078,
+                            0.000685318314936012,
+                            0.0006897522835060954,
+                            0.0006745737046003342,
+                            0.0008671511313878,
+                            0.0005519991391338408,
+                            0.0006618749466724694,
+                            0.0005458846571855247,
+                            0.00048387900460511446,
+                            0.0007630230393260717,
+                            0.0006800986593589187,
+                            0.0008858009823597968,
+                            0.0005791778676211834,
+                            0.0006878529093228281,
+                            0.0005736556486226618,
+                            0.0006958959274925292,
+                            0.0005982575821690261,
+                            0.000762462499551475,
+                            0.0006404005107469857,
+                            0.000635067408438772,
+                            0.0005458136438392103,
+                            0.0006299325614236295,
+                            0.0007780585438013077,
+                            0.0006350981420837343,
+                            0.0006981721380725503,
+                            0.0006858370616100729,
+                            0.0007322267629206181,
+                            0.000676373194437474,
+                            0.0007953306194394827,
+                            0.0006307433941401541,
+                            0.0008275146246887743,
+                            0.0008437120704911649,
+                            0.0006277310312725604,
+                            0.0005602853489108384,
+                            0.0007491149590350688,
+                            0.0005637961439788342,
+                            0.0005567622720263898,
+                            0.0006818743422627449,
+                            0.0005596668925136328,
+                            0.0007516398327425122,
+                            0.0008923675050027668,
+                            0.0005307461833581328,
+                            0.0006239814683794975,
+                            0.0008536897366866469,
+                            0.0006792545318603516,
+                            0.0007546377019025385,
+                            0.0006529715610668063,
+                            0.0007806713692843914,
+                            0.0005899200914427638,
+                            0.0006436925032176077,
+                            0.0006707524880766869,
+                            0.0006891887751407921,
+                            0.0008247719961218536,
+                            0.0005420551169663668,
+                            0.0007585940766148269,
+                            0.0005056567606516182,
+                            0.0006273529725149274,
+                            0.0006361496634781361,
+                            0.0008495243382640183,
+                            0.0005489684408530593,
+                            0.0005853826296515763,
+                            0.0010158214718103409,
+                            0.0005268929526209831,
+                            0.0007200529798865318,
+                            0.0006558540626429021,
+                            0.000579203013330698,
+                            0.0006989951361902058,
+                            0.0006177483592182398,
+                            0.0006545177893713117,
+                            0.0006882884190417826,
+                            0.0005208624061197042,
+                            0.0006457005511038005,
+                            0.0005805999971926212,
+                            0.0006072535179555416,
+                            0.000545910676009953,
+                            0.0006128970417194068,
+                            0.000848128052894026,
+                            0.0005733900470659137,
+                            0.0008798990165814757,
+                            0.0005851430469192564,
+                            0.0005699312314391136,
+                            0.0006180619238875806,
+                            0.0005868153530173004,
+                            0.0006960578029975295,
+                            0.0005756508326157928,
+                            0.0006258602952584624,
+                            0.0005538301193155348,
+                            0.0007450940902344882,
+                            0.0006307870498858392,
+                            0.000794046267401427,
+                            0.0009879007702693343,
+                            0.0006135146832093596,
+                            0.0006669914000667632,
+                            0.0006375054363161325,
+                            0.0007001437479630113,
+                            0.0008885313291102648,
+                            0.0006951698451302946,
+                            0.0007675081142224371,
+                            0.0007251625647768378,
+                            0.0008073966600932181,
+                            0.0005343231605365872,
+                            0.0006284642149694264,
+                            0.0007123541436158121,
+                            0.0007113225292414427,
+                            0.0005585308535955846,
+                            0.0007186323055066168,
+                            0.0005616678390651941,
+                            0.000519298599101603,
+                            0.0005188987124711275,
+                            0.0008174533140845597,
+                            0.000585067318752408,
+                            0.0008624405600130558,
+                            0.0006932024261914194,
+                            0.0007241056300699711,
+                            0.0005001922836527228,
+                            0.0009658299386501312,
+                            0.0009356974042020738,
+                            0.0008339310879819095,
+                            0.0008770613349042833,
+                            0.0005397885688580573,
+                            0.000654027156997472,
+                            0.0005952991778030992,
+                            0.0009590803529135883,
+                            0.0006544219795614481,
+                            0.0006086894427426159,
+                            0.0006121068727225065,
+                            0.0005634804256260395,
+                            0.0007282226579263806,
+                            0.0007198702660389245,
+                            0.0005828057182952762,
+                            0.0005328857223503292,
+                            0.0007026649545878172,
+                            0.0008111760253086686,
+                            0.0005880408571101725,
+                            0.0008082150598056614,
+                            0.000603282474912703,
+                            0.0005894370842725039,
+                            0.0006544762291014194,
+                            0.0005581185687333345,
+                            0.0005382022354751825,
+                            0.0006299472879618406,
+                            0.0005796220502816141,
+                            0.0006960437167435884,
+                            0.0006803303840570152,
+                            0.0006581313209608197,
+                            0.0007307803607545793,
+                            0.0006524991476908326,
+                            0.000744774763006717,
+                            0.0007316555711440742,
+                            0.0006335731013678014,
+                            0.0006880099535919726,
+                            0.0005729876575060189,
+                            0.0007120543741621077,
+                            0.0007705009775236249,
+                            0.0007111153099685907,
+                            0.0008696348522789776,
+                            0.0007834286079742014,
+                            0.0006550253601744771,
+                            0.0005225851782597601,
+                            0.0006870944635011256,
+                            0.0006884306785650551,
+                            0.000538310210686177,
+                            0.0006675293552689254,
+                            0.0006625925307162106,
+                            0.0006851208745501935,
+                            0.0007393662817776203,
+                            0.0006722515681758523,
+                            0.0006313672056421638,
+                            0.0007959870854392648,
+                            0.0006965865613892674,
+                            0.0007854976574890316,
+                            0.0007069694111123681,
+                            0.0006137393647804856,
+                            0.0007953871972858906,
+                            0.0006062981556169689,
+                            0.000659091689158231,
+                            0.0006285891868174076,
+                            0.0006560960318893194,
+                            0.0006916429265402257,
+                            0.0006010629585944116,
+                            0.000736426911316812,
+                            0.0006628857809118927,
+                            0.0007446226663887501,
+                            0.0006100477185100317,
+                            0.0007395163411274552,
+                            0.0006065674242563546,
+                            0.0005919054383412004,
+                            0.0005797483609057963,
+                            0.0006090310052968562,
+                            0.0006369802285917103,
+                            0.0007840744801796973,
+                            0.0005868742009624839,
+                            0.0005754205631092191,
+                            0.0006436168914660811,
+                            0.0006033495301380754,
+                            0.0007026543607935309,
+                            0.0008043245761655271,
+                            0.0004646973393391818,
+                            0.0006250901496969163,
+                            0.000840126711409539,
+                            0.0006691362941637635,
+                            0.0006620686035603285,
+                            0.0006513337139040232,
+                            0.0007223783759400249,
+                            0.0006987908273003995,
+                            0.0006568178650923073,
+                            0.0006276639178395271,
+                            0.0005993758677504957,
+                            0.0008049038588069379,
+                            0.0006771857151761651,
+                            0.000783290306571871,
+                            0.0006926496862433851,
+                            0.0005814451724290848,
+                            0.0006639367202296853,
+                            0.0005864058039151132,
+                            0.0005991045036353171,
+                            0.0005855110357515514,
+                            0.0005290028639137745,
+                            0.0005147529300302267,
+                            0.0007395457359962165,
+                            0.0005778304184786975,
+                            0.0005548865301534534,
+                            0.0006234872271306813,
+                            0.0006487570353783667,
+                            0.000709205458406359,
+                            0.0007514870958402753,
+                            0.0007112747407518327,
+                            0.0005839085788466036,
+                            0.0005512398784048855,
+                            0.0008062022388912737,
+                            0.0006205611862242222,
+                            0.0006157937459647655,
+                            0.0005683376220986247,
+                            0.000528498727362603,
+                            0.0006282631657086313,
+                            0.0006457690033130348,
+                            0.0006140651530586183,
+                            0.0008199061849154532,
+                            0.0006063692271709442,
+                            0.0005357928457669914,
+                            0.0006644355598837137,
+                            0.0005871307221241295,
+                            0.0006711893365718424,
+                            0.000748171005398035,
+                            0.000897470919881016,
+                            0.0005314225563779473,
+                            0.0006231723236851394,
+                            0.0007336622220464051,
+                            0.0005888452287763357,
+                            0.0005059823160991073,
+                            0.0005919403047300875,
+                            0.0005923212738707662,
+                            0.000610017275903374,
+                            0.0005493346834555268,
+                            0.0005464256391860545,
+                            0.0006561221089214087,
+                            0.0006119360332377255,
+                            0.0006124101928435266,
+                            0.0005832392489537597,
+                            0.0006713264738209546,
+                            0.0005658958107233047,
+                            0.0006784903234802186,
+                            0.0006214246968738735,
+                            0.0007618213421665132,
+                            0.0005853910697624087,
+                            0.0005020078970119357,
+                            0.0006822888390161097,
+                            0.0005816708435304463,
+                            0.0006998809985816479,
+                            0.0005479714600369334,
+                            0.0007094445754773915,
+                            0.0006602485664188862,
+                            0.0005996683030389249,
+                            0.000524098111782223,
+                            0.0005685118376277387,
+                            0.000580237596295774,
+                            0.0005929211620241404,
+                            0.0005699815228581429,
+                            0.0005989912315271795,
+                            0.0006247975979931653,
+                            0.0007713324157521129,
+                            0.0005695961881428957,
+                            0.0004915387253277004,
+                            0.0006252087187021971,
+                            0.0005598803400062025,
+                            0.0008721251506358385,
+                            0.0006710960296913981,
+                            0.0005095057422295213,
+                            0.0006180724012665451,
+                            0.0006652228184975684,
+                            0.000627668749075383,
+                            0.0005478265229612589,
+                            0.0005036313668824732,
+                            0.0005494546494446695,
+                            0.0007449849508702755,
+                            0.0005456910002976656,
+                            0.0006595461745746434,
+                            0.0005328715778887272,
+                            0.0007176389335654676,
+                            0.0005346039542928338,
+                            0.0005939114489592612,
+                            0.0005791683215647936,
+                            0.0005150494980625808,
+                            0.0007619585958309472,
+                            0.0005874659982509911,
+                            0.0007237080717459321,
+                            0.0005701895570382476,
+                            0.0007436384330503643,
+                            0.0006636413163505495,
+                            0.0005233145784586668,
+                            0.0005688016535714269,
+                            0.0005702088237740099,
+                            0.0006837683613412082,
+                            0.0008504709112457931,
+                            0.0005976335378363729,
+                            0.0005880796816200018,
+                            0.0005712254787795246,
+                            0.0006099981837905943,
+                            0.0005178721039555967,
+                            0.0005958789261057973,
+                            0.000532466103322804,
+                            0.0005277961608953774,
+                            0.000624479609541595,
+                            0.0006227884441614151,
+                            0.0006486921338364482,
+                            0.0005551663925871253,
+                            0.0004937071353197098,
+                            0.0006402888102456927,
+                            0.0006781606934964657,
+                            0.0005884733982384205,
+                            0.0006953871343284845,
+                            0.0005683443159796298,
+                            0.0006035788101144135,
+                            0.0007036193273961544,
+                            0.0005945308366790414,
+                            0.0005137354019097984,
+                            0.0005415756604634225,
+                            0.0008602017187513411,
+                            0.0004997037467546761,
+                            0.000625209417194128,
+                            0.0005600800504907966,
+                            0.000611710362136364,
+                            0.0005417375359684229,
+                            0.0005172647652216256,
+                            0.000509413075633347,
+                            0.0006469886866398156,
+                            0.0005811406881548464,
+                            0.0006036388804204762,
+                            0.0005426001152954996,
+                            0.0006054426776245236,
+                            0.0006930335075594485,
+                            0.000740537594538182,
+                            0.0008995882817544043,
+                            0.0005967066972516477,
+                            0.0006795970839448273,
+                            0.0007293906528502703,
+                            0.0005487513262778521,
+                            0.0005531269125640392,
+                            0.000607689842581749,
+                            0.0005719177424907684,
+                            0.0006252533057704568,
+                            0.0008150748326443136,
+                            0.000627276545856148,
+                            0.0007140813977457583,
+                            0.0005795526085421443,
+                            0.0005622225580736995,
+                            0.0007171924225986004,
+                            0.0006461242446675897,
+                            0.0006553744315169752,
+                            0.0006217550253495574,
+                            0.0006592892459593713,
+                            0.000581081083510071,
+                            0.0005861669778823853,
+                            0.0006971153197810054,
+                            0.0005889413296245039,
+                            0.0006861697183921933,
+                            0.0006054233526811004,
+                            0.0005697779124602675,
+                            0.0006568589597009122,
+                            0.0005518709658645093,
+                            0.0005531258648261428,
+                            0.0005856403731741011,
+                            0.0004755034751724452,
+                            0.0005229228991083801,
+                            0.0006388655165210366,
+                            0.0005391603917814791,
+                            0.0006992561393417418,
+                            0.0006652367301285267,
+                            0.0006452760426327586,
+                            0.0007473828736692667,
+                            0.0008092672796919942,
+                            0.0006376019446179271,
+                            0.0006008164491504431,
+                            0.000617433397565037,
+                            0.000648150104098022,
+                            0.000588340510148555,
+                            0.0006503624026663601,
+                            0.0004857908352278173,
+                            0.0005246930522844195,
+                            0.0005388717399910092,
+                            0.0006216595065779984,
+                            0.0006673202151432633,
+                            0.000661153462715447,
+                            0.0005105300806462765,
+                            0.0005949382903054357,
+                            0.0007178304949775338,
+                            0.0007877742755226791,
+                            0.0007720048888586462,
+                            0.0006932428223080933,
+                            0.0005747752729803324,
+                            0.0005285735242068768,
+                            0.0006704998086206615,
+                            0.0006128678214736283,
+                            0.0006348572787828743,
+                            0.0006686770357191563,
+                            0.0007063763914629817,
+                            0.0007547865970991552,
+                            0.0006025522598065436,
+                            0.0005009874003008008,
+                            0.0006587780080735683,
+                            0.000649783993139863,
+                            0.0005687130033038557,
+                            0.0007799587328918278,
+                            0.0006419199053198099,
+                            0.0006266448763199151,
+                            0.0006184580852277577,
+                            0.00052466313354671,
+                            0.0006131596164777875,
+                            0.000547403993550688,
+                            0.0006692127208225429,
+                            0.0005330952699296176,
+                            0.0006100250757299364,
+                            0.0005813782918266952,
+                            0.0007280844147317111,
+                            0.000743023818358779,
+                            0.0006362017011269927,
+                            0.000754108012188226,
+                            0.0005676464643329382,
+                            0.0005599136929959059,
+                            0.0009825826855376363,
+                            0.0006056907586753368,
+                            0.0006534714484587312,
+                            0.0005639040027745068,
+                            0.0006462637684307992,
+                            0.0005109942867420614,
+                            0.0005230248789303005,
+                            0.0005575475515797734,
+                            0.0005796661134809256,
+                            0.0005955785163678229,
+                            0.0006534988642670214,
+                            0.0005885857390239835,
+                            0.0007496018661186099,
+                            0.0006719148950651288,
+                            0.0007283134618774056,
+                            0.0006873203674331307,
+                            0.0006664696848019958,
+                            0.000684866274241358,
+                            0.0005367954145185649,
+                            0.0005727275856770575,
+                            0.0006004690076224506,
+                            0.0006452128873206675,
+                            0.000524776813108474,
+                            0.000629428424872458,
+                            0.0007022032514214516,
+                            0.0005412432365119457,
+                            0.0005721286288462579,
+                            0.0006470907828770578,
+                            0.0006129221874289215,
+                            0.0005712111014872789,
+                            0.0005684140487574041,
+                            0.0005607677740044892,
+                            0.000599140184931457,
+                            0.0007904758094809949,
+                            0.0006498786387965083,
+                            0.0005653102416545153,
+                            0.0005654689157381654,
+                            0.0006749875610694289,
+                            0.0006941596511751413,
+                            0.000529807060956955,
+                            0.0006730578024871647,
+                            0.0007125756819732487,
+                            0.0006720294477418065,
+                            0.0008432136964984238,
+                            0.00059973553288728,
+                            0.0006642465596087277,
+                            0.0004866346425842494,
+                            0.0005980202695354819,
+                            0.0007811951218172908,
+                            0.0005951839848421514,
+                            0.0006708418950438499,
+                            0.0004959647194482386,
+                            0.0007333428948186338,
+                            0.0006403594743460417,
+                            0.0005422258982434869,
+                            0.0005658029695041478,
+                            0.0005461860564537346,
+                            0.0008111631614156067,
+                            0.0006780769326724112,
+                            0.0005592143279500306,
+                            0.0005891192704439163,
+                            0.0010554438922554255,
+                            0.0005695535219274461,
+                            0.0006703418912366033,
+                            0.0007882342324592173,
+                            0.0006666591507382691,
+                            0.0008487139130011201,
+                            0.0006075982237234712,
+                            0.0005869818851351738,
+                            0.0005855531780980527,
+                            0.0006506992504000664,
+                            0.000662658829241991,
+                            0.0007211421034298837,
+                            0.0006171525456011295,
+                            0.0006454003741964698,
+                            0.000783274183049798,
+                            0.0006820980343036354,
+                            0.0007518632337450981,
+                            0.0005106426542624831,
+                            0.0008585222531110048,
+                            0.0007263462175615132,
+                            0.0006479781004600227,
+                            0.0004974609473720193,
+                            0.0006508977385237813,
+                            0.0005490464973263443,
+                            0.0005910285981371999,
+                            0.0006436428520828485,
+                            0.0006831174832768738,
+                            0.0005562799633480608,
+                            0.0008499773102812469,
+                            0.000668211025185883,
+                            0.0008125741151161492,
+                            0.0007264958694577217,
+                            0.0005991553771309555,
+                            0.000800740672275424,
+                            0.0005517647368833423,
+                            0.0005954041262157261,
+                            0.000726409547496587,
+                            0.0005492923664860427,
+                            0.0006700052763335407,
+                            0.0005386308184824884,
+                            0.0008333731675520539,
+                            0.0005630107480101287,
+                            0.0006264170515350997,
+                            0.0006204803939908743,
+                            0.0006410732166841626,
+                            0.000582852924708277,
+                            0.0006079024169594049,
+                            0.0007900252821855247,
+                            0.0007063684170134366,
+                            0.0008082050480879843,
+                            0.0006673031020909548,
+                            0.0007313800742849708,
+                            0.0005399053334258497,
+                            0.0006742716650478542,
+                            0.0005834407638758421,
+                            0.0008452333859167993,
+                            0.0006337526720017195,
+                            0.0005692370468750596,
+                            0.0008128376794047654,
+                            0.0006255467305891216,
+                            0.00057429657317698,
+                            0.0005817983183078468,
+                            0.0006551317637786269,
+                            0.0005965065211057663,
+                            0.0007610657485201955,
+                            0.0005797573830932379,
+                            0.0006367106689140201,
+                            0.0007317745476029813,
+                            0.0005642174510285258,
+                            0.0006460775039158762,
+                            0.0006594613078050315,
+                            0.0006181860808283091,
+                            0.0007128330762498081,
+                            0.0007435299921780825,
+                            0.0006047298084013164,
+                            0.0007408233359456062,
+                            0.0006526671350002289,
+                            0.0005409237346611917,
+                            0.0006822917493991554,
+                            0.0006522966432385147,
+                            0.0006322735571302474,
+                            0.0005516481469385326,
+                            0.0008404835825785995,
+                            0.0007039743941277266,
+                            0.0007319269934669137,
+                            0.0007336301496252418,
+                            0.0006750613101758063,
+                            0.0006426644395105541,
+                            0.000668831926304847,
+                            0.0006192454020492733,
+                            0.0006179825868457556,
+                            0.0006008109776303172,
+                            0.0005338070332072675,
+                            0.000528693781234324,
+                            0.0005903783021494746,
+                            0.000617724028415978,
+                            0.000609114533290267,
+                            0.0007015938754193485,
+                            0.0008937501115724444,
+                            0.0006777563830837607,
+                            0.0008007526048459113,
+                            0.0006111983675509691,
+                            0.0006670824368484318,
+                            0.0006215969915501773,
+                            0.000668663065880537,
+                            0.0005358757916837931,
+                            0.0007717370172031224,
+                            0.0006909345393069088,
+                            0.0006187246181070805,
+                            0.0007131517631933093,
+                            0.0008153126109391451,
+                            0.0006859633722342551,
+                            0.0006172192515805364,
+                            0.0005773062002845109,
+                            0.0006405891617760062,
+                            0.0006641115760430694,
+                            0.0007474329322576523,
+                            0.0006213982123881578,
+                            0.0007658495451323688,
+                            0.0006855868850834668,
+                            0.0007154237246140838,
+                            0.0005584424361586571,
+                            0.0005476734950207174,
+                            0.0007742147427052259,
+                            0.0005739993066526949,
+                            0.0007275033858604729,
+                            0.000818976724985987,
+                            0.0005883817793801427,
+                            0.0006069082301110029,
+                            0.0005588865606114268,
+                            0.0005770793068222702,
+                            0.0005582683370448649,
+                            0.0006083762855269015,
+                            0.0005764866946265101,
+                            0.0005886786966584623,
+                            0.0006029533105902374,
+                            0.0006089790258556604,
+                            0.0005639430019073188,
+                            0.0006767233717255294,
+                            0.0006478187860921025,
+                            0.0005598037387244403,
+                            0.0005726650124415755,
+                            0.0005423874827101827,
+                            0.0005728219985030591,
+                            0.0007196879014372826,
+                            0.0005829078145325184,
+                            0.0005742064095102251,
+                            0.0006224576500244439,
+                            0.0006970677641220391,
+                            0.0005606272025033832,
+                            0.0004946925328113139,
+                            0.0006257587228901684,
+                            0.0006636379403062165,
+                            0.0007404116331599653,
+                            0.0007018493488430977,
+                            0.0007182308472692966,
+                            0.0005254881689324975,
+                            0.0008654938428662717,
+                            0.000574000587221235,
+                            0.0005180974840186536,
+                            0.000669717148412019,
+                            0.0007947242702357471,
+                            0.0007217902457341552,
+                            0.0005726591916754842,
+                            0.0006414243252947927,
+                            0.0005287591484375298,
+                            0.000549514836166054,
+                            0.0006380796548910439,
+                            0.0005261097103357315,
+                            0.0005430998280644417,
+                            0.0006731287576258183,
+                            0.0005536929820664227,
+                            0.0005116768879815936,
+                            0.0006285848212428391,
+                            0.0006259418441914022,
+                            0.0006055132835172117,
+                            0.000549567281268537,
+                            0.0006721393438056111,
+                            0.0007190268952399492,
+                            0.0005998199339956045,
+                            0.000581251282710582,
+                            0.000544820330105722,
+                            0.0005977240507490933,
+                            0.0005401897942647338,
+                            0.00059767987113446,
+                            0.0007116654887795448,
+                            0.0006127811502665281,
+                            0.0008871469763107598,
+                            0.0005116479005664587,
+                            0.0005246206419542432,
+                            0.0007368192891590297,
+                            0.0006585201481357217,
+                            0.0006095995777286589,
+                            0.0005496187950484455,
+                            0.0006137490272521973,
+                            0.0006531000835821033,
+                            0.0006152130081318319,
+                            0.0005968694458715618,
+                            0.0006208564154803753,
+                            0.0005458135856315494,
+                            0.000597877602558583,
+                            0.0005952272331342101,
+                            0.000576322665438056,
+                            0.0005052281776443124,
+                            0.0006139188772067428,
+                            0.0006147547392174602,
+                            0.0005226093344390392,
+                            0.0005639927694573998,
+                            0.00048271645209752023,
+                            0.0005671193357557058,
+                            0.0007914308807812631,
+                            0.0005532618379220366,
+                            0.0006182134384289384,
+                            0.0006635675090365112,
+                            0.0006020404398441315,
+                            0.0005431690951809287,
+                            0.0005380366346798837,
+                            0.0005706604570150375,
+                            0.0005999168497510254,
+                            0.0006585392402485013,
+                            0.0006364938453771174,
+                            0.0005382017698138952,
+                            0.0007888972759246826,
+                            0.0006339930114336312,
+                            0.0006299129454419017,
+                            0.0006293256883509457,
+                            0.0005326881655491889,
+                            0.0005459677777253091,
+                            0.0005849194130860269,
+                            0.0005986755713820457,
+                            0.0005795125616714358,
+                            0.0005742507055401802,
+                            0.0005610781954601407,
+                            0.0005576884723268449,
+                            0.0005454032798297703,
+                            0.0005453990888781846,
+                            0.0005396822234615684,
+                            0.0005287356907501817,
+                            0.0005156404222361743,
+                            0.0005721488851122558,
+                            0.0005797623307444155,
+                            0.00061917764833197,
+                            0.0006248194258660078,
+                            0.0005504863802343607,
+                            0.0006709533627144992,
+                            0.000714255846105516,
+                            0.0007142723770812154,
+                            0.0006249676807783544,
+                            0.0005900808027945459,
+                            0.0005666049546562135,
+                            0.0005908970488235354,
+                            0.000650547503028065,
+                            0.0004893894656561315,
+                            0.0005897024529986084,
+                            0.0006660722428932786,
+                            0.0005790364812128246,
+                            0.0005066054873168468,
+                            0.0005500539555214345,
+                            0.0005560868885368109,
+                            0.000590676034335047,
+                            0.0005586117040365934,
+                            0.0006553042330779135,
+                            0.0005610619555227458,
+                            0.0006458769203163683,
+                            0.000583114568144083,
+                            0.0005963367875665426,
+                            0.0007038268377073109,
+                            0.0005370283615775406,
+                            0.0005374641623347998,
+                            0.0007229179609566927,
+                            0.00045306916581466794,
+                            0.0005995311075821519,
+                            0.0006404168670997024,
+                            0.0007235637749545276,
+                            0.0005200671148486435,
+                            0.0006416359101422131,
+                            0.000574591220356524,
+                            0.000642321421764791,
+                            0.000505804899148643,
+                            0.000542902504093945,
+                            0.0006863344460725784,
+                            0.0005299411714076996,
+                            0.0005838452489115298,
+                            0.000566616770811379,
+                            0.0006726285209879279,
+                            0.0005677161389030516,
+                            0.000641864666249603,
+                            0.0007063423981890082,
+                            0.000590324925724417,
+                            0.0006185650709085166,
+                            0.0004789796657860279,
+                            0.000569177616853267,
+                            0.0005219885497353971,
+                            0.0006167270475998521,
+                            0.0005652409163303673,
+                            0.0007530731381848454,
+                            0.0005651613464578986,
+                            0.0005931671475991607,
+                            0.0006368071190081537,
+                            0.0005060039693489671,
+                            0.0004925319808535278,
+                            0.0006751689943484962,
+                            0.000567074166610837,
+                            0.0007400649483315647,
+                            0.0005770450225099921,
+                            0.00059285550378263,
+                            0.0005638116272166371,
+                            0.0005969097255729139,
+                            0.0006881369627080858,
+                            0.0007687958423048258,
+                            0.0007762082968838513,
+                            0.0006450010696426034,
+                            0.0005189783405512571,
+                            0.0005135165993124247,
+                            0.0005540938000194728,
+                            0.0005533318035304546,
+                            0.0007555109332315624,
+                            0.0007007019594311714,
+                            0.0007168938172981143,
+                            0.0005378241185098886,
+                            0.0006781893316656351,
+                            0.0005918418173678219,
+                            0.0006934016710147262,
+                            0.0007987501448951662,
+                            0.0006466840277425945,
+                            0.0006079331506043673,
+                            0.0005672990810126066,
+                            0.0005783913657069206,
+                            0.0005447253934107721,
+                            0.0006800000555813313,
+                            0.000612666888628155,
+                            0.0006013328675180674,
+                            0.0007159433444030583,
+                            0.0006972952396608889,
+                            0.0005682603805325925,
+                            0.0006043657194823027,
+                            0.0005810254951938987,
+                            0.0007119654910638928,
+                            0.0006801732233725488,
+                            0.0005512231145985425,
+                            0.0006007864139974117,
+                            0.0005985983298160136,
+                            0.0006943322368897498,
+                            0.000503257499076426,
+                            0.0006024176254868507,
+                            0.0005780724459327757,
+                            0.0005994871025905013,
+                            0.0007759731961414218,
+                            0.0007795124547556043,
+                            0.0006241752998903394,
+                            0.000577411032281816,
+                            0.0005141416331753135,
+                            0.0006873184465803206,
+                            0.0005505731678567827,
+                            0.0007325840997509658,
+                            0.0006796239758841693,
+                            0.0005986266769468784,
+                            0.00048772981972433627,
+                            0.0007324585458263755,
+                            0.0005708477110601962,
+                            0.0006239822832867503,
+                            0.0005799327045679092,
+                            0.0005586213665083051,
+                            0.0006612578872591257,
+                            0.0005984351737424731,
+                            0.0005951423663645983,
+                            0.0006329894531518221,
+                            0.0005411074380390346,
+                            0.0008127845358103514,
+                            0.0004964111722074449,
+                            0.0006575945299118757,
+                            0.0005955194355919957,
+                            0.0006304277922026813,
+                            0.0007054122979752719,
+                            0.00062078662449494,
+                            0.0006341523840092123,
+                            0.0005662626936100423,
+                            0.0005695437430404127,
+                            0.0006284804549068213,
+                            0.0007127979770302773,
+                            0.000504286668729037,
+                            0.000652612536214292,
+                            0.0005541237769648433,
+                            0.0007703323499299586,
+                            0.0005832623573951423,
+                            0.0007022185600362718,
+                            0.0005430672317743301,
+                            0.0005816358025185764,
+                            0.0006385095766745508,
+                            0.0005826266133226454,
+                            0.0005660583265125751,
+                            0.0007462407229468226,
+                            0.0005181169835850596,
+                            0.00045230973046272993,
+                            0.0006325892172753811,
+                            0.0005732935969717801,
+                            0.000594173208810389,
+                            0.0006591689307242632,
+                            0.00056224234867841,
+                            0.0006268154247663915,
+                            0.000663386657834053,
+                            0.0006708114524371922,
+                            0.0006795938825234771,
+                            0.0006300202803686261,
+                            0.00048763357335701585,
+                            0.0005910336039960384,
+                            0.0006300759851001203,
+                            0.0005414971383288503,
+                            0.0006864358438178897,
+                            0.0007056383183225989,
+                            0.0005678922170773149,
+                            0.0005432430771179497,
+                            0.0006034303805790842,
+                            0.0006368767935782671,
+                            0.0005161635344848037,
+                            0.0006064855842851102,
+                            0.0006491127423942089,
+                            0.0006283049006015062,
+                            0.0005356637993827462,
+                            0.0005244644125923514,
+                            0.0005861747777089477,
+                            0.00049142527859658,
+                            0.0005426128045655787,
+                            0.000672412570565939,
+                            0.0005136143881827593,
+                            0.0006485719350166619,
+                            0.0006143411155790091,
+                            0.0005068614846095443,
+                            0.0006965282373130322,
+                            0.0006909876829013228,
+                            0.00068361108424142,
+                            0.0007552470196969807,
+                            0.0007923100492917001,
+                            0.00059681007405743,
+                            0.0005507457535713911,
+                            0.0006319745443761349,
+                            0.0006199385388754308,
+                            0.00048440942191518843,
+                            0.0005649795057252049,
+                            0.0007271965150721371,
+                            0.0005971803911961615,
+                            0.0005261618061922491,
+                            0.0006120709585957229,
+                            0.0006134050199761987,
+                            0.0006819313275627792,
+                            0.0005532422219403088,
+                            0.0005938736139796674,
+                            0.0005150299402885139,
+                            0.0007902944344095886,
+                            0.0005626805359497666,
+                            0.0005624286131933331,
+                            0.0005959614063613117,
+                            0.0005916372174397111,
+                            0.0007650808547623456,
+                            0.0006168312393128872,
+                            0.0005589720676653087,
+                            0.0006299238302744925,
+                            0.0007620270480401814,
+                            0.0005555040552280843,
+                            0.0005434148479253054,
+                            0.0005083800060674548,
+                            0.000711638480424881,
+                            0.000543570495210588,
+                            0.0005801932420581579,
+                            0.0005212452379055321,
+                            0.0007679928676225245,
+                            0.0006285697454586625,
+                            0.0005608639330603182,
+                            0.0007400033064186573,
+                            0.0005797708290629089,
+                            0.0005356124020181596,
+                            0.0006016285042278469,
+                            0.0005273270653560758,
+                            0.0008167013875208795,
+                            0.000662431528326124,
+                            0.000647964421659708,
+                            0.000523167138453573,
+                            0.0006221835501492023,
+                            0.0006135968142189085,
+                            0.00047215481754392385,
+                            0.0005182127351872623,
+                            0.0006692251772619784,
+                            0.0006278754444792867,
+                            0.0006422398728318512,
+                            0.000604169094003737,
+                            0.0005053278291597962,
+                            0.0005667682853527367,
+                            0.000672468391712755,
+                            0.0005571819492615759,
+                            0.0006916550337336957,
+                            0.0006382686551660299,
+                            0.0005693132989108562,
+                            0.0005887893494218588,
+                            0.0005522681749425828,
+                            0.0005930030602030456,
+                            0.0005366529803723097,
+                            0.0005044647841714323,
+                            0.0006340771797113121,
+                            0.0006390686030499637,
+                            0.0006116819567978382,
+                            0.0005893745110370219,
+                            0.0005092892679385841,
+                            0.0005183999892324209,
+                            0.000703170895576477,
+                            0.0006394024239853024,
+                            0.000509306148160249,
+                            0.0005324036465026438,
+                            0.0006441393634304404,
+                            0.0006825461750850081,
+                            0.0006147847161628306,
+                            0.0006060219602659345,
+                            0.0004938666825182736,
+                            0.0006271114107221365,
+                            0.0005666647339239717,
+                            0.0007664085715077817,
+                            0.0008481365512125194,
+                            0.0006288699223659933,
+                            0.0005655970307998359,
+                            0.0005331114516593516,
+                            0.000698962714523077,
+                            0.0006138908211141825,
+                            0.0007829971727915108,
+                            0.0006703294930048287,
+                            0.0007656575180590153,
+                            0.000579117564484477,
+                            0.0005029517924413085,
+                            0.0006102282204665244,
+                            0.0009093566332012415,
+                            0.0007684907759539783,
+                            0.0004824948264285922,
+                            0.0005544135929085314,
+                            0.0005914552602916956,
+                            0.0010835136054083705,
+                            0.0005831517046317458,
+                            0.0006216798792593181,
+                            0.0005302084027789533,
+                            0.0007734708487987518,
+                            0.0008369102724827826,
+                            0.0005473062628880143,
+                            0.0005623264005407691,
+                            0.0006016908446326852,
+                            0.0005805233377031982,
+                            0.0005460986867547035,
+                            0.0007706147152930498,
+                            0.0006179196061566472,
+                            0.0006128086824901402,
+                            0.0006264730473048985,
+                            0.0005386702832765877,
+                            0.0007148370495997369,
+                            0.0004928408307023346,
+                            0.0006408652407117188,
+                            0.0008272358682006598,
+                            0.0006459358846768737,
+                            0.0005557821714319289,
+                            0.0007400285103358328,
+                            0.0006044873152859509,
+                            0.0006522389012388885,
+                            0.0008887380245141685,
+                            0.000600636238232255,
+                            0.0005437590298242867,
+                            0.0006803456926718354,
+                            0.0006317216902971268,
+                            0.0005769324488937855,
+                            0.0006645666435360909,
+                            0.0005254083662293851,
+                            0.0006335080834105611,
+                            0.000624713662546128,
+                            0.0005823370302096009,
+                            0.000710383290424943,
+                            0.0006342540145851672,
+                            0.0005444977432489395,
+                            0.0005871528410352767,
+                            0.0005323257064446807,
+                            0.0005101062124595046,
+                            0.0006692077731713653,
+                            0.0005239059100858867,
+                            0.0006099824677221477,
+                            0.0006564187933690846,
+                            0.00047626852756366134,
+                            0.0005411456804722548,
+                            0.0006003966554999352,
+                            0.0005536109674721956,
+                            0.0006613578298129141,
+                            0.0006198033224791288,
+                            0.0005111969076097012,
+                            0.0005705808289349079,
+                            0.0005677341250702739,
+                            0.0005532009527087212,
+                            0.0004707435437012464,
+                            0.0005981071735732257,
+                            0.0008010092424228787,
+                            0.0006252509192563593,
+                            0.0005728654796257615,
+                            0.0006486726924777031,
+                            0.0006054767873138189,
+                            0.000576443038880825,
+                            0.0005194354453124106,
+                            0.0006638565682806075,
+                            0.0006468063220381737,
+                            0.0005188147188164294,
+                            0.0005899531533941627,
+                            0.0005572487134486437,
+                            0.0005260388134047389,
+                            0.0006092022522352636,
+                            0.0005287048406898975,
+                            0.0004838396853301674,
+                            0.0005484596476890147,
+                            0.00061991618713364,
+                            0.0005871854373253882,
+                            0.0006086325738579035,
+                            0.0007342550088651478,
+                            0.0005875074421055615,
+                            0.0006010645884089172,
+                            0.0006000252906233072,
+                            0.000538633088581264,
+                            0.0006920788437128067,
+                            0.0008565072785131633,
+                            0.0006801139679737389,
+                            0.0005275671719573438,
+                            0.000635045173112303,
+                            0.0005684344796463847,
+                            0.0005984168965369463,
+                            0.0005732566351071,
+                            0.0005926508456468582,
+                            0.0006851030047982931,
+                            0.0005324827507138252,
+                            0.0005792133160866797,
+                            0.0008099984261207283,
+                            0.0005874861963093281,
+                            0.0006346105947159231,
+                            0.00048018331290222704,
+                            0.0006004798342473805,
+                            0.0006132329581305385,
+                            0.0006745619466528296,
+                            0.0006456216215156019,
+                            0.0005100492853671312,
+                            0.00056154077174142,
+                            0.0006068803486414254,
+                            0.0006627778057008982,
+                            0.0006434830138459802,
+                            0.0005891431355848908,
+                            0.0005399004439823329,
+                            0.0006405518506653607,
+                            0.0006316294311545789,
+                            0.0005861532990820706,
+                            0.0006618503830395639,
+                            0.000556468206923455,
+                            0.0005974766099825501,
+                            0.0007061132346279919,
+                            0.0006759953103028238,
+                            0.0005484196590259671,
+                            0.0006931688985787332,
+                            0.0005845365230925381,
+                            0.0006451652152463794,
+                            0.0006846171454526484,
+                            0.000553123012650758,
+                            0.0007357274880632758,
+                            0.000532867677975446,
+                            0.0006482893950305879,
+                            0.000602701329626143,
+                            0.0006399893318302929,
+                            0.0007486924296244979,
+                            0.000543196510989219,
+                            0.0005378880887292325,
+                            0.0005159665597602725,
+                            0.0007355064153671265,
+                            0.0007469544652849436,
+                            0.0006066134665161371,
+                            0.0005763806402683258,
+                            0.0006448996136896312,
+                            0.0005170647636987269,
+                            0.0005230632959865034,
+                            0.0006803471478633583,
+                            0.0006019638385623693,
+                            0.0006543277995660901,
+                            0.0007105107069946826,
+                            0.0008260916220024228,
+                            0.0005947429453954101,
+                            0.0005659608286805451,
+                            0.0005667432560585439,
+                            0.0005977442488074303,
+                            0.0005593542591668665,
+                            0.0006126394728198647,
+                            0.0005342127406038344,
+                            0.0006073505501262844,
+                            0.0007524843676947057,
+                            0.0006183329969644547,
+                            0.0007106735720299184,
+                            0.0007095844484865665,
+                            0.0005709228571504354,
+                            0.0005872330511920154,
+                            0.0004921832587569952,
+                            0.0006100804312154651,
+                            0.0005736751481890678,
+                            0.0005868977750651538,
+                            0.0005916988593526185,
+                            0.000642523227725178,
+                            0.0006345764850266278,
+                            0.0006935126148164272,
+                            0.0009049623622559011,
+                            0.0006116407457739115,
+                            0.0006369745824486017,
+                            0.0006417289259843528,
+                            0.0006983962957747281,
+                            0.0006232499144971371,
+                            0.0005636396817862988,
+                            0.000565746275242418,
+                            0.0005769866402260959,
+                            0.0005859936936758459,
+                            0.0006298825610429049,
+                            0.0005118070403113961,
+                            0.0008083482971414924,
+                            0.0005838506622239947,
+                            0.0006333673372864723,
+                            0.0005341568030416965,
+                            0.0005589560605585575,
+                            0.0006312817567959428,
+                            0.0005459649837575853,
+                            0.0006366222514770925,
+                            0.0005459252861328423,
+                            0.0005585246835835278,
+                            0.0006520965253002942,
+                            0.0007117707864381373,
+                            0.0005398321663960814,
+                            0.000652286980766803,
+                            0.0006431485526263714,
+                            0.0007130694575607777,
+                            0.0006558249006047845,
+                            0.0006381655111908913,
+                            0.0005455860518850386,
+                            0.0007250059861689806,
+                            0.0005378694622777402,
+                            0.0006673326715826988,
+                            0.0007129068253561854,
+                            0.0005901779513806105,
+                            0.000576032092794776,
+                            0.0007396519649773836,
+                            0.0006225854158401489,
+                            0.0007742394227534533,
+                            0.0006582316709682345,
+                            0.0005931108607910573,
+                            0.0005671106628142297,
+                            0.000598232145421207,
+                            0.0007505587418563664,
+                            0.000570646720007062,
+                            0.0005968357436358929,
+                            0.0006990459514781833,
+                            0.000752352352719754,
+                            0.0005926490994170308,
+                            0.0006327021401375532,
+                            0.0007310655782930553,
+                            0.000678101321682334,
+                            0.0005802718224003911,
+                            0.0005680879112333059,
+                            0.0005297015886753798,
+                            0.0007512483280152082,
+                            0.0006332125631161034,
+                            0.0006355606601573527,
+                            0.0005230474635027349,
+                            0.0005937325186096132,
+                            0.00048374233301728964,
+                            0.0006009953795000911,
+                            0.000550526543520391,
+                            0.0006446169572882354,
+                            0.0005873616901226342,
+                            0.0005917777889408171,
+                            0.0005687368102371693,
+                            0.000635814038105309,
+                            0.0006295684142969549,
+                            0.0005613588145934045,
+                            0.0006802261923439801,
+                            0.000648842251393944,
+                            0.0005141622968949378,
+                            0.0006027831695973873,
+                            0.0005973837105557323,
+                            0.0006259896908886731,
+                            0.0005793580203317106,
+                            0.000571935495827347,
+                            0.0005788216949440539,
+                            0.0005929881008341908,
+                            0.0005351270083338022,
+                            0.0005186517373658717,
+                            0.0005923189455643296,
+                            0.0005423448164947331,
+                            0.0006380484555847943,
+                            0.0005765718524344265,
+                            0.0007023827638477087,
+                            0.0008445670828223228,
+                            0.0005600823787972331,
+                            0.0006036408594809473,
+                            0.0005848673754371703,
+                            0.0005961192073300481,
+                            0.0005576020921580493,
+                            0.0006759427487850189,
+                            0.0005902735865674913,
+                            0.0006028416100889444,
+                            0.0006587416282854974,
+                            0.0005828804569318891,
+                            0.0005064660217612982,
+                            0.0005242250044830143,
+                            0.0005842418759129941,
+                            0.000608972564805299,
+                            0.0005666229990310967,
+                            0.0006023265304975212,
+                            0.0005942067946307361,
+                            0.0004713178495876491,
+                            0.0006419479614123702,
+                            0.0005468893796205521,
+                            0.0006105299689806998,
+                            0.0005405802512541413,
+                            0.0005925454315729439,
+                            0.000570472446270287,
+                            0.0006352714844979346,
+                            0.0006082617910578847,
+                            0.0006116440636105835,
+                            0.000528214150108397,
+                            0.0006504428456537426,
+                            0.0006218164344318211,
+                            0.0006655114120803773,
+                            0.0006374376825988293,
+                            0.0006746239378117025,
+                            0.0006164423539303243,
+                            0.0007559140212833881,
+                            0.0006491764797829092,
+                            0.0005818275967612863,
+                            0.0005696858861483634,
+                            0.0006129242246970534,
+                            0.0005926925805397332,
+                            0.00048069143667817116,
+                            0.0005513436626642942,
+                            0.0005637156427837908,
+                            0.0006393917719833553,
+                            0.000608082686085254,
+                            0.0005870201275683939,
+                            0.0005693463026545942,
+                            0.0005750266136601567,
+                            0.0005110578495077789,
+                            0.00062303472077474,
+                            0.0005933906650170684,
+                            0.0005839813966304064,
+                            0.0007172158220782876,
+                            0.000599469116423279,
+                            0.000665063678752631,
+                            0.0006375739467330277,
+                            0.0005787470145151019,
+                            0.0007110493606887758,
+                            0.0006131120026111603,
+                            0.0006429383647628129,
+                            0.0006365806912072003,
+                            0.0005748969851993024,
+                            0.000645842810627073,
+                            0.0006397020188160241,
+                            0.0005546692409552634,
+                            0.0006563017959706485,
+                            0.00061870674835518,
+                            0.0007232865318655968,
+                            0.0005489095929078758,
+                            0.0005092633073218167,
+                            0.0007219988037832081,
+                            0.0005685920477844775,
+                            0.0005682789487764239,
+                            0.0007472166907973588,
+                            0.0005363072268664837,
+                            0.0005650892271660268,
+                            0.000571253418456763,
+                            0.0005354360328055918,
+                            0.0005593992536887527,
+                            0.0007115610642358661,
+                            0.0005416954518295825,
+                            0.0006537159788422287,
+                            0.0005267568049021065,
+                            0.0006208782433532178,
+                            0.0005463886773213744,
+                            0.0007269212510436773,
+                            0.0005536448443308473,
+                            0.0007630784530192614,
+                            0.0005695910076610744,
+                            0.0006304874550551176,
+                            0.0007454032311215997,
+                            0.0005260746111162007,
+                            0.0005045532598160207,
+                            0.0006259732181206346,
+                            0.0005308261606842279,
+                            0.0005930043407715857,
+                            0.0006585242808796465,
+                            0.000597991282120347,
+                            0.000687610125169158,
+                            0.0006460145814344287,
+                            0.0005702167400158942,
+                            0.000544582144357264,
+                            0.0005592762026935816,
+                            0.000630957365501672,
+                            0.000555974489543587,
+                            0.000533096375875175,
+                            0.0007357794092968106,
+                            0.0005602694000117481,
+                            0.0005704142968170345,
+                            0.0005396740161813796,
+                            0.000546176393982023,
+                            0.0005983197479508817,
+                            0.0006750166066922247,
+                            0.0005847321590408683,
+                            0.0005938825197517872,
+                            0.0005869138985872269,
+                            0.0005897423252463341,
+                            0.0005612733657471836,
+                            0.0006622800719924271,
+                            0.0005090047488920391,
+                            0.0005706515512429178,
+                            0.0005883992998860776,
+                            0.0006063241744413972,
+                            0.000634901924058795,
+                            0.000572187127545476,
+                            0.0006504073971882463,
+                            0.0007172406185418367,
+                            0.0006002283771522343,
+                            0.0007091955631040037,
+                            0.0006915144040249288,
+                            0.0005446970462799072,
+                            0.0006031726370565593,
+                            0.0005999664426781237,
+                            0.0006286950665526092,
+                            0.0005969498888589442,
+                            0.000733032647985965,
+                            0.000604841101448983,
+                            0.0006016505649313331,
+                            0.0006335871876217425,
+                            0.0006051213131286204,
+                            0.0005265421350486577,
+                            0.0006922228494659066,
+                            0.0005649715312756598,
+                            0.0006879995344206691,
+                            0.0006384248845279217,
+                            0.0006660391809418797,
+                            0.0006200551870279014,
+                            0.0005644247867166996,
+                            0.0005766687099821866,
+                            0.0006107276421971619,
+                            0.0005824370891787112,
+                            0.0006046282942406833,
+                            0.0006867318297736347,
+                            0.000549910357221961,
+                            0.0006361219566315413,
+                            0.0006391064962372184,
+                            0.0005861931131221354,
+                            0.000735627778340131,
+                            0.0005381762166507542,
+                            0.0005946542951278389,
+                            0.0006754571804776788,
+                            0.0008429535664618015,
+                            0.0006582397618331015,
+                            0.0005698441527783871,
+                            0.0006544964271597564,
+                            0.000513086502905935,
+                            0.0006146098603494465,
+                            0.000645600666757673,
+                            0.0006424649618566036,
+                            0.0005864746053703129,
+                            0.0006286255083978176,
+                            0.0007535834447480738,
+                            0.0005253301351331174,
+                            0.0006310101016424596,
+                            0.0006308127194643021,
+                            0.0007210290059447289,
+                            0.0006146664381958544,
+                            0.0007275125244632363,
+                            0.000655330833978951,
+                            0.000686893705278635,
+                            0.0005348238628357649,
+                            0.0006504379562102258,
+                            0.0006421825964935124,
+                            0.0005336642498150468,
+                            0.0005528094479814172,
+                            0.0008066347218118608,
+                            0.0004949449794366956,
+                            0.0005662787007167935,
+                            0.0005173711106181145,
+                            0.00054334377637133,
+                            0.000547053525224328,
+                            0.0005464684218168259,
+                            0.0005773225566372275,
+                            0.0006114622228778899,
+                            0.0005641485913656652,
+                            0.0007235227385535836,
+                            0.0006063556647859514,
+                            0.0006361579871736467,
+                            0.0006657710764557123,
+                            0.0006181743810884655,
+                            0.0006617569597437978,
+                            0.0006238706409931183,
+                            0.000671209127176553,
+                            0.0006464701727963984,
+                            0.0006743731210008264,
+                            0.000594107317738235,
+                            0.000685693696141243,
+                            0.0006573416758328676,
+                            0.0005259501631371677,
+                            0.0005596649716608226,
+                            0.0007396658766083419,
+                            0.0005166884511709213,
+                            0.0008237719885073602,
+                            0.0008919793181121349,
+                            0.0006157637108117342,
+                            0.0008023973787203431,
+                            0.0007153096375986934,
+                            0.0005335196037776768,
+                            0.0006660051294602454,
+                            0.0005652834079228342,
+                            0.0005543937440961599,
+                            0.0006031737430021167,
+                            0.0006504043703898787,
+                            0.0005493879434652627,
+                            0.000544221606105566,
+                            0.0005024080164730549,
+                            0.0007791048847138882,
+                            0.0005761173670180142,
+                            0.000620451697614044,
+                            0.0006246204720810056,
+                            0.0005700261099264026,
+                            0.0005839581717737019,
+                            0.0005774219171144068,
+                            0.0006560285110026598,
+                            0.0005998670822009444,
+                            0.00048627538490109146,
+                            0.0005585287581197917,
+                            0.00063946598675102,
+                            0.0006017702398821712,
+                            0.0006599981570616364,
+                            0.0005109792109578848,
+                            0.0006144765065982938,
+                            0.000603902735747397,
+                            0.000733980443328619,
+                            0.0005682818591594696,
+                            0.0006361306295730174,
+                            0.0006301415851339698,
+                            0.0006452332017943263,
+                            0.000731071166228503,
+                            0.000839386775624007,
+                            0.0007343200268223882,
+                            0.0006520803435705602,
+                            0.00082502665463835,
+                            0.0006989403045736253,
+                            0.000681117526255548,
+                            0.0006776171503588557,
+                            0.0007043130462989211,
+                            0.0008050482720136642,
+                            0.0008278132299892604,
+                            0.0008622031309641898,
+                            0.0008190266671590507,
+                            0.0006150174303911626,
+                            0.0005161170847713947,
+                            0.0006385812303051353,
+                            0.0006538884481415153,
+                            0.000760619412176311,
+                            0.0006059726001694798,
+                            0.0008553360239602625,
+                            0.000520840345416218,
+                            0.0008085284498520195,
+                            0.0006789002218283713,
+                            0.0007412873674184084,
+                            0.0005834897747263312,
+                            0.0006378808175213635,
+                            0.0006470162770710886,
+                            0.000801718735601753,
+                            0.0006839769193902612,
+                            0.0005756759783253074,
+                            0.0006082238396629691,
+                            0.0007311804802156985,
+                            0.0006533840787597001,
+                            0.0005760465865023434,
+                            0.0007692449144087732,
+                            0.0006652509910054505,
+                            0.0005956611130386591,
+                            0.0005651787505485117,
+                            0.0007872721180319786,
+                            0.0006641795043833554,
+                            0.0005730444681830704,
+                            0.0006505619385279715,
+                            0.000771518680267036,
+                            0.0006613936275243759,
+                            0.0006663246895186603,
+                            0.0006552110426127911,
+                            0.0005451716133393347,
+                            0.0007135288324207067,
+                            0.0007698655244894326,
+                            0.0006836648099124432,
+                            0.0006335360230877995,
+                            0.0005644542397931218,
+                            0.0006068720831535757,
+                            0.0007557024946436286,
+                            0.0006147052627056837,
+                            0.0008736841264180839,
+                            0.0008202025783248246,
+                            0.0006426876643672585,
+                            0.0006084005581215024,
+                            0.0006894024554640055,
+                            0.0005443673580884933,
+                            0.0005492568598128855,
+                            0.0006564611685462296,
+                            0.0008770393906161189,
+                            0.0006919354782439768,
+                            0.0006972383707761765,
+                            0.0005956765962764621,
+                            0.0005864164559170604,
+                            0.0007883123471401632,
+                            0.00076683284714818,
+                            0.0007240414852276444,
+                            0.0005680908798240125,
+                            0.0005876848008483648,
+                            0.0005891780601814389,
+                            0.0006529565434902906,
+                            0.0006785579607822001,
+                            0.0007095455075614154,
+                            0.0006912162061780691,
+                            0.0005912650376558304,
+                            0.0008333214209415019,
+                            0.0006528738304041326,
+                            0.00073525175685063,
+                            0.0006792224012315273,
+                            0.0006609937991015613,
+                            0.0007097458583302796,
+                            0.0006937432335689664,
+                            0.0006472557433880866,
+                            0.0005103567382320762,
+                            0.0007122692186385393,
+                            0.0006911138771101832,
+                            0.0005270215915516019,
+                            0.0006454963004216552,
+                            0.000946789572481066,
+                            0.0005510036135092378,
+                            0.0009106300421990454,
+                            0.0007154113845899701,
+                            0.0005728784599341452,
+                            0.0006329895695671439,
+                            0.0005385370459407568,
+                            0.0008311342680826783,
+                            0.0005341116921044886,
+                            0.00047704996541142464,
+                            0.0007354331319220364,
+                            0.0008874668274074793,
+                            0.0007249711779877543,
+                            0.0007383818156085908,
+                            0.0005218774313107133,
+                            0.0005913955974392593,
+                            0.000566092086955905,
+                            0.0006175313610583544,
+                            0.0006995680159889162,
+                            0.0005604324978776276,
+                            0.0007567238644696772,
+                            0.0007437351741828024,
+                            0.0006533890264108777,
+                            0.0006758599192835391,
+                            0.0005866988212801516,
+                            0.0006049484945833683,
+                            0.0007439822074957192,
+                            0.0005550539353862405,
+                            0.0010308631462976336,
+                            0.0009156978339888155,
+                            0.000894675322342664,
+                            0.0007095563923940063,
+                            0.000735914392862469,
+                            0.000621921441052109,
+                            0.0007801719475537539,
+                            0.0007672535721212626,
+                            0.0006529162055812776,
+                            0.0005694018327631056,
+                            0.0007104240939952433,
+                            0.0006519460002891719,
+                            0.0008162357262335718,
+                            0.0007390807149931788,
+                            0.0007165950373746455,
+                            0.000538008171133697,
+                            0.0005641325260512531,
+                            0.0007816630532033741,
+                            0.0007504766690544784,
+                            0.0009056617273017764,
+                            0.0005480810068547726,
+                            0.0012079718289896846,
+                            0.0008153219823725522,
+                            0.0005955591332167387,
+                            0.0005783364758826792,
+                            0.0005765723763033748,
+                            0.0009821192361414433,
+                            0.0006202154327183962,
+                            0.0006572370766662061,
+                            0.0006959070451557636,
+                            0.0006600841297768056,
+                            0.000614270509686321,
+                            0.0007553138420917094,
+                            0.000812369049526751,
+                            0.0012267674319446087,
+                            0.0006070053204894066,
+                            0.0006276029744185507,
+                            0.0007441142806783319,
+                            0.0005938649410381913,
+                            0.0006324452115222812,
+                            0.0008382883970625699,
+                            0.0006763748242519796,
+                            0.0008259480237029493,
+                            0.0012379602994769812,
+                            0.0007095568580552936,
+                            0.0005446990835480392,
+                            0.0005598905845545232,
+                            0.0007252614013850689,
+                            0.0007808447699062526,
+                            0.000684864295180887,
+                            0.0006333409110084176,
+                            0.0005279714823700488,
+                            0.0008978985715657473,
+                            0.0006400736165232956,
+                            0.0006764728459529579,
+                            0.0007511359290219843,
+                            0.000709777872543782,
+                            0.0010715060634538531,
+                            0.0006815141532570124,
+                            0.0006433936068788171,
+                            0.0005843217368237674,
+                            0.0005724793882109225,
+                            0.0006435405230149627,
+                            0.0006151372217573225,
+                            0.0005287375533953309,
+                            0.0006198788178153336,
+                            0.0006256273481994867,
+                            0.0005676366854459047,
+                            0.0006048082723282278,
+                            0.0005484071443788707,
+                            0.0006740742246620357,
+                            0.0007600566605105996,
+                            0.0008611576049588621,
+                            0.0006192440632730722,
+                            0.000578406557906419,
+                            0.0008025518618524075,
+                            0.0007904909434728324,
+                            0.0006388005567714572,
+                            0.0007970198639668524,
+                            0.0005638676811940968,
+                            0.000799904577434063,
+                            0.0007921149372123182,
+                            0.0008180965087376535,
+                            0.0008769398555159569,
+                            0.0006153159192763269,
+                            0.0009271784801967442,
+                            0.0005698804161511362,
+                            0.0012624325463548303,
+                            0.0007104541873559356,
+                            0.0005981921567581594,
+                            0.0007868235115893185,
+                            0.0006530693499371409,
+                            0.0007448364049196243,
+                            0.0007162329275161028,
+                            0.0006981892511248589,
+                            0.000575927784666419,
+                            0.000558515777811408,
+                            0.0007914258167147636,
+                            0.0005600526346825063,
+                            0.000880377774592489,
+                            0.000733009772375226,
+                            0.0008721252670511603,
+                            0.0006634616875089705,
+                            0.0006663025706075132,
+                            0.0011895999778062105,
+                            0.0006250237347558141,
+                            0.000658206467051059,
+                            0.0005561388679780066,
+                            0.0007057127077132463,
+                            0.0011419818038120866,
+                            0.0007323173922486603,
+                            0.0008458371157757938,
+                            0.0006667912239208817,
+                            0.0008092376519925892,
+                            0.0005278601893223822,
+                            0.0007290499052032828,
+                            0.0006529543898068368,
+                            0.000817138294223696,
+                            0.0005762320943176746,
+                            0.0008587527554482222,
+                            0.0007055678288452327,
+                            0.0005508313188329339,
+                            0.0007203951827250421,
+                            0.0006496206624433398,
+                            0.0006211906438693404,
+                            0.000557798775844276,
+                            0.0006818624096922576,
+                            0.0005907548475079238,
+                            0.0007501878426410258,
+                            0.000848851224873215,
+                            0.0005692772683687508,
+                            0.0005037763621658087,
+                            0.0006030236254446208,
+                            0.0006362527492456138,
+                            0.0006531276158057153,
+                            0.0005725109367631376,
+                            0.0005984273739159107,
+                            0.0006773427594453096,
+                            0.0006085861241444945,
+                            0.0006747236475348473,
+                            0.000740592775400728,
+                            0.0007059689960442483,
+                            0.000697610666975379,
+                            0.0005723137292079628,
+                            0.0006321488763205707,
+                            0.0009865780593827367,
+                            0.0005331597640179098,
+                            0.0005474483477883041,
+                            0.0005734085571020842,
+                            0.00070690963184461,
+                            0.0006016596453264356,
+                            0.0005067008314654231,
+                            0.000596154248341918,
+                            0.0006752422195859253,
+                            0.0006305996794253588,
+                            0.0005785984103567898,
+                            0.0008002331596799195,
+                            0.0006181426579132676,
+                            0.0005610828520730138,
+                            0.0005605390761047602,
+                            0.0005202264292165637,
+                            0.000890857249032706,
+                            0.00046151955029927194,
+                            0.0004862906353082508,
+                            0.000530136632733047,
+                            0.0008177284034900367,
+                            0.0008342239889316261,
+                            0.0006152731948532164,
+                            0.0006382595165632665,
+                            0.0005925923469476402,
+                            0.0005923433927819133,
+                            0.0009029436623677611,
+                            0.00066982681164518,
+                            0.000842027657199651,
+                            0.0008130725473165512,
+                            0.000714786583557725,
+                            0.0007567773573100567,
+                            0.0007016338058747351,
+                            0.0007634250796400011,
+                            0.0006355365039780736,
+                            0.0006517263827845454,
+                            0.0005412682075984776,
+                            0.0005430597811937332,
+                            0.0006165679660625756,
+                            0.0006733078043907881,
+                            0.0006287091528065503,
+                            0.000545828661415726,
+                            0.0005741624045185745,
+                            0.0005619211005978286,
+                            0.0007173094782046974,
+                            0.0006542199989780784,
+                            0.0005688824458047748,
+                            0.0006711464957334101,
+                            0.000746336707379669,
+                            0.0006994763971306384,
+                            0.000810060475487262,
+                            0.0005126024479977787,
+                            0.0006636013858951628,
+                            0.0007452010759152472,
+                            0.0008014567429199815,
+                            0.0007481398642994463,
+                            0.0007724697352387011,
+                            0.0007091853185556829,
+                            0.0006328510935418308,
+                            0.0005345271783880889,
+                            0.0006305845454335213,
+                            0.0011417192872613668,
+                            0.0006087291985750198,
+                            0.0005811316077597439,
+                            0.0006362333078868687,
+                            0.000945333915296942,
+                            0.0005492187920026481,
+                            0.000567429990042001,
+                            0.0006236334447748959,
+                            0.0006863620365038514,
+                            0.000800435256678611,
+                            0.0006087160436436534,
+                            0.0005469251773320138,
+                            0.0006677900091744959,
+                            0.0005273184506222606,
+                            0.0006846640026196837,
+                            0.000615153054241091,
+                            0.0005201433086767793,
+                            0.000583143497351557,
+                            0.0005970990750938654,
+                            0.0005724345683120191,
+                            0.0008969765622168779,
+                            0.0005999038694426417,
+                            0.0009145273943431675,
+                            0.0006532214465551078,
+                            0.0006752366898581386,
+                            0.0005568309570662677,
+                            0.0008725464576855302,
+                            0.0006024108733981848,
+                            0.0006002489244565368,
+                            0.0005582463345490396,
+                            0.0009616936440579593,
+                            0.000804870156571269,
+                            0.0005422909744083881,
+                            0.0007352244574576616,
+                            0.0007366085192188621,
+                            0.0010206253500655293,
+                            0.0006477443967014551,
+                            0.0005840232479386032,
+                            0.0005856226780451834,
+                            0.0006977977463975549,
+                            0.0007310215150937438,
+                            0.0005841991514898837,
+                            0.0007956149056553841,
+                            0.0005665556527674198,
+                            0.0005595435504801571,
+                            0.0007777570281177759,
+                            0.0005765053210780025,
+                            0.0006409810157492757,
+                            0.0005857279757037759,
+                            0.0006462908932007849,
+                            0.0005981232970952988,
+                            0.0007293531089089811,
+                            0.0007647491875104606,
+                            0.0007417873130179942,
+                            0.0005743714864365757,
+                            0.0005120259593240917,
+                            0.0007602659752592444,
+                            0.0006339570973068476,
+                            0.0007045597303658724,
+                            0.0007166761788539588,
+                            0.0005799237405881286,
+                            0.0004995337803848088,
+                            0.0005609040963463485,
+                            0.0007125037373043597,
+                            0.0006732781184837222,
+                            0.0008257806184701622,
+                            0.0005781569634564221,
+                            0.0006307489820756018,
+                            0.0006644200184382498,
+                            0.0006584706716239452,
+                            0.000662074948195368,
+                            0.0008222722099162638,
+                            0.0010314029641449451,
+                            0.0005811620503664017,
+                            0.0007378812879323959,
+                            0.000614291406236589,
+                            0.00060234748525545,
+                            0.0005132899968884885,
+                            0.0006646336405538023,
+                            0.0005581555888056755,
+                            0.0005430817254818976,
+                            0.0006250148289836943,
+                            0.0007843695930205286,
+                            0.0005485091242007911,
+                            0.0006092936382628977,
+                            0.0009236181504093111,
+                            0.0007574282353743911,
+                            0.0006380101549439132,
+                            0.0007918610353954136,
+                            0.0005651875399053097,
+                            0.0006721857935190201,
+                            0.0005707088275812566,
+                            0.0005299624754115939,
+                            0.0005879107629880309,
+                            0.00072306307265535,
+                            0.0005400885711424053,
+                            0.0004983245744369924,
+                            0.0005373570020310581,
+                            0.0006481914897449315,
+                            0.0009577564196661115,
+                            0.0006579346372745931,
+                            0.000680483237374574,
+                            0.0006765997386537492,
+                            0.0006108785746619105,
+                            0.0005303389043547213,
+                            0.000584082561545074,
+                            0.0005229274975135922,
+                            0.000555651553440839,
+                            0.0005665775970555842,
+                            0.0005942272837273777,
+                            0.0009784265421330929,
+                            0.00056221813429147,
+                            0.0005572348600253463,
+                            0.0005540134152397513,
+                            0.000730851897969842,
+                            0.0007597676012665033,
+                            0.0005504805594682693,
+                            0.0005713280406780541,
+                            0.0006140417535789311,
+                            0.0007118033827282488,
+                            0.0006061974563635886,
+                            0.0005591096123680472,
+                            0.0007044537924230099,
+                            0.0007296918774954975,
+                            0.0006350676994770765,
+                            0.0005706561496481299,
+                            0.0006428437773138285,
+                            0.001236425363458693,
+                            0.0008055549114942551,
+                            0.000647278327960521,
+                            0.0005779885104857385,
+                            0.0005851613823324442,
+                            0.0006163789075799286,
+                            0.000551482371520251,
+                            0.0007364950724877417,
+                            0.0006571593112312257,
+                            0.0006439651479013264,
+                            0.0005610011867247522,
+                            0.0006613726727664471,
+                            0.0005876118084415793,
+                            0.0006268302095122635,
+                            0.0006186963291838765,
+                            0.0005630816449411213,
+                            0.0007424615905620158,
+                            0.0005906766164116561,
+                            0.0005698577733710408,
+                            0.0005380004877224565,
+                            0.0006012823432683945,
+                            0.0005674983840435743,
+                            0.0008990711066871881,
+                            0.0006337842787615955,
+                            0.0005292142741382122,
+                            0.0005441943649202585,
+                            0.0009040961740538478,
+                            0.0006906298804096878,
+                            0.0006613389123231173,
+                            0.0008576674736104906,
+                            0.000521951587870717,
+                            0.0007436235318891704,
+                            0.0005194478435441852,
+                            0.0005188683280721307,
+                            0.0005910156178288162,
+                            0.0006943331682123244,
+                            0.0006107406807132065,
+                            0.0005845926352776587,
+                            0.0005096166860312223,
+                            0.0005685922224074602,
+                            0.0006752763874828815,
+                            0.0004948413115926087,
+                            0.0007106551202014089,
+                            0.0005744831287302077,
+                            0.0006247006822377443,
+                            0.0005628240760415792,
+                            0.0007053997251205146
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 65,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "5": {
+                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type_is_module": false,
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 66,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.0178313497453928
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 63,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.03605125471949577
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 67,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.03753050044178963
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "6": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.V_linears.2",
+                "input_tensor_infos": [
+                    {
+                        "id": 67,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.03753050044178963
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0009660659707151353,
+                            0.0011442751856520772,
+                            0.0007612574263475835,
+                            0.0008692672126926482,
+                            0.0010626731673255563,
+                            0.0009725427371449769,
+                            0.0008318920736201108,
+                            0.0006666609551757574,
+                            0.0007731810910627246,
+                            0.0008125475142151117,
+                            0.000755240092985332,
+                            0.0010041220812126994,
+                            0.0010187070583924651,
+                            0.0007083381060510874,
+                            0.000954583054408431,
+                            0.001056770677678287,
+                            0.0010391727555543184,
+                            0.000741548603400588,
+                            0.0009708382422104478,
+                            0.0008089213515631855,
+                            0.001636917470023036,
+                            0.0010975472396239638,
+                            0.0009172142599709332,
+                            0.0007518500206060708,
+                            0.0011374854948371649,
+                            0.0008029888849705458,
+                            0.0008310948614962399,
+                            0.0008601916488260031,
+                            0.001097365515306592,
+                            0.0010764974867925048,
+                            0.0009506437927484512,
+                            0.0007587355794385076,
+                            0.0009179480839520693,
+                            0.0008885532151907682,
+                            0.0007236570818349719,
+                            0.0009122893679887056,
+                            0.0011227618670091033,
+                            0.0009025842300616205,
+                            0.000752463995013386,
+                            0.0011004264233633876,
+                            0.0009216371108777821,
+                            0.0008568795747123659,
+                            0.001378354849293828,
+                            0.001073598861694336,
+                            0.001291248481720686,
+                            0.0011810354189947248,
+                            0.0013038960751146078,
+                            0.0010995958000421524,
+                            0.0015819986583665013,
+                            0.0012178681790828705,
+                            0.0010232499334961176,
+                            0.0012450467329472303,
+                            0.0011503007262945175,
+                            0.0008899884996935725,
+                            0.0013220340479165316,
+                            0.0008987797191366553,
+                            0.0013739197747781873,
+                            0.0008657812722958624,
+                            0.0009303070255555212,
+                            0.0010168402222916484,
+                            0.0009758378146216273,
+                            0.0011500808177515864,
+                            0.001110681681893766,
+                            0.001108460477553308,
+                            0.000788894307333976,
+                            0.0008616465493105352,
+                            0.0008702299674041569,
+                            0.0010774822439998388,
+                            0.0009387718746438622,
+                            0.0009472413221374154,
+                            0.0011876230128109455,
+                            0.0008239565650001168,
+                            0.0008098922553472221,
+                            0.001112013473175466,
+                            0.0009446240146644413,
+                            0.0008608935168012977,
+                            0.0007273750961758196,
+                            0.0014382144436240196,
+                            0.001155039295554161,
+                            0.0009668697603046894,
+                            0.001282314071431756,
+                            0.001118003623560071,
+                            0.0008778993505984545,
+                            0.0009329775348305702,
+                            0.0008326112874783576,
+                            0.001127326744608581,
+                            0.0008217347203753889,
+                            0.0008569115307182074,
+                            0.0009119560709223151,
+                            0.0012076967395842075,
+                            0.0008367120753973722,
+                            0.0010840775212273002,
+                            0.00102999375667423,
+                            0.0009593790164217353,
+                            0.000863795168697834,
+                            0.0008217436261475086,
+                            0.0008033281774260104,
+                            0.0007334434194490314,
+                            0.000930356269236654,
+                            0.0009086891077458858,
+                            0.0011849256698042154,
+                            0.0008225526544265449,
+                            0.0014033053303137422,
+                            0.0008458830416202545,
+                            0.0008779282215982676,
+                            0.0007783101173117757,
+                            0.0010164579143747687,
+                            0.0010066835675388575,
+                            0.0008950994815677404,
+                            0.0019366267370060086,
+                            0.0009570478578098118,
+                            0.0007685025339014828,
+                            0.0008607744821347296,
+                            0.0012156913289800286,
+                            0.000828379881568253,
+                            0.001171377138234675,
+                            0.0009034478571265936,
+                            0.0008539220434613526,
+                            0.0009735526982694864,
+                            0.0007895345333963633,
+                            0.0011364843230694532,
+                            0.0013755224645137787,
+                            0.0008869994780980051,
+                            0.0008695201249793172,
+                            0.0010022013448178768,
+                            0.0007466473034583032,
+                            0.0011584678431972861,
+                            0.0010096709011122584,
+                            0.0008726149098947644,
+                            0.0008504724828526378,
+                            0.0011981260031461716,
+                            0.000875424942933023,
+                            0.0009226285619661212,
+                            0.0012703979155048728,
+                            0.001383312395773828,
+                            0.0010592172620818019,
+                            0.001378616550937295,
+                            0.000992576009593904,
+                            0.0011165180476382375,
+                            0.0008450133027508855,
+                            0.0010030405828729272,
+                            0.0010353754041716456,
+                            0.0008551864884793758,
+                            0.0008935758378356695,
+                            0.0011223737383261323,
+                            0.000799902540165931,
+                            0.0007656140951439738,
+                            0.0010185138089582324,
+                            0.0009993963176384568,
+                            0.0008241729810833931,
+                            0.0011264703935012221,
+                            0.0010328961070626974,
+                            0.0015453753294423223,
+                            0.0009359702235087752,
+                            0.0010474077425897121,
+                            0.0013793350663036108,
+                            0.0011860617669299245,
+                            0.0010221449192613363,
+                            0.001111300545744598,
+                            0.0007673654472455382,
+                            0.0007885234081186354,
+                            0.0012640539789572358,
+                            0.0008837651112116873,
+                            0.0010995124466717243,
+                            0.0010841445764526725,
+                            0.0009408923797309399,
+                            0.0011930747423321009,
+                            0.0008685165084898472,
+                            0.0009210510761477053,
+                            0.0009359720279462636,
+                            0.0008289979305118322,
+                            0.000964004488196224,
+                            0.0017526609590277076,
+                            0.0008936912054196,
+                            0.0010068394476547837,
+                            0.0007483783992938697,
+                            0.0007685534656047821,
+                            0.0014559892006218433,
+                            0.0009755899664014578,
+                            0.0009052230161614716,
+                            0.0008587630582042038,
+                            0.000890166440512985,
+                            0.0008431465248577297,
+                            0.0007868953398428857,
+                            0.0013426230289041996,
+                            0.0006413085502572358,
+                            0.0014084667200222611,
+                            0.0011346940882503986,
+                            0.0007931242580525577,
+                            0.0008849127334542572,
+                            0.0008833138272166252,
+                            0.0008324701339006424,
+                            0.0009823838481679559,
+                            0.001012208522297442,
+                            0.00106389494612813,
+                            0.0007092657033354044,
+                            0.0010222693672403693,
+                            0.000982969650067389,
+                            0.001037504174746573,
+                            0.001060999114997685,
+                            0.0010535760084167123,
+                            0.0011691931867972016,
+                            0.0011829979484900832,
+                            0.001003017881885171,
+                            0.0011448649456724524,
+                            0.0010113936150446534,
+                            0.0009293774492107332,
+                            0.001287655089981854,
+                            0.0008657390717417002,
+                            0.0012315681669861078,
+                            0.0013914882438257337,
+                            0.0008514108485542238,
+                            0.0009210575371980667,
+                            0.001725776819512248,
+                            0.0010460559278726578,
+                            0.0008264250936917961,
+                            0.000996601884253323,
+                            0.001151761505752802,
+                            0.0010042744688689709,
+                            0.001148711540736258,
+                            0.0013792235404253006,
+                            0.0009637982584536076,
+                            0.001185687491670251,
+                            0.0009122072951868176,
+                            0.0008543660514988005,
+                            0.0008174680988304317,
+                            0.0011393961030989885,
+                            0.0014542052522301674,
+                            0.0010936796898022294,
+                            0.001277497736737132,
+                            0.0011937514645978808,
+                            0.0008805563556961715,
+                            0.001065750140696764,
+                            0.0010173289338126779,
+                            0.00119857769459486,
+                            0.0009435965330339968,
+                            0.001160103245638311,
+                            0.0010038811014965177,
+                            0.0010796975111588836,
+                            0.001364024356007576,
+                            0.0009020412690006196,
+                            0.0009177796891890466,
+                            0.0013853103155270219,
+                            0.000912887800950557,
+                            0.0008106670575216413,
+                            0.0010440306505188346,
+                            0.001276221126317978,
+                            0.0010244569275528193,
+                            0.0011713157873600721,
+                            0.0011654506670311093,
+                            0.0008098372491076589,
+                            0.0007885413360781968,
+                            0.0013177935034036636,
+                            0.0008066902519203722,
+                            0.0009905440965667367,
+                            0.0010847228113561869,
+                            0.0008950253250077367,
+                            0.0009254456963390112,
+                            0.0008886992000043392,
+                            0.0007063334924168885,
+                            0.0010166227584704757,
+                            0.0009312802576459944,
+                            0.0012710472801700234,
+                            0.0010960750514641404,
+                            0.000749165948946029,
+                            0.0009104394121095538,
+                            0.0009640812641009688,
+                            0.0008149768691509962,
+                            0.0007648526225239038,
+                            0.0011512566125020385,
+                            0.0012301539536565542,
+                            0.0009278676589019597,
+                            0.00113798794336617,
+                            0.0011355179594829679,
+                            0.001137029379606247,
+                            0.0007868445827625692,
+                            0.0008441171958111227,
+                            0.0011440594680607319,
+                            0.0009276222554035485,
+                            0.0009056112612597644,
+                            0.0014140139101073146,
+                            0.0011118081165477633,
+                            0.0008515705121681094,
+                            0.0008276893640868366,
+                            0.0012297440553084016,
+                            0.0010172543115913868,
+                            0.0008635163540020585,
+                            0.0009706126293167472,
+                            0.0009437443222850561,
+                            0.0008647213689982891,
+                            0.0007607999723404646,
+                            0.0008442917023785412,
+                            0.0009129677200689912,
+                            0.0013515345053747296,
+                            0.0009185663075186312,
+                            0.0009612661087885499,
+                            0.0009884813334792852,
+                            0.0009519196464680135,
+                            0.001033807173371315,
+                            0.0009499195730313659,
+                            0.0009527401998639107,
+                            0.0012735117925330997,
+                            0.001044888747856021,
+                            0.0010982571402564645,
+                            0.001070697559043765,
+                            0.0008801936055533588,
+                            0.0011377218179404736,
+                            0.0008683732012286782,
+                            0.0010705377208068967,
+                            0.0007899126503616571,
+                            0.0009644555393606424,
+                            0.000895217526704073,
+                            0.001083925599232316,
+                            0.0008713320712558925,
+                            0.0009794291108846664,
+                            0.0009489567019045353,
+                            0.000911818235181272,
+                            0.0009609925327822566,
+                            0.0012212179135531187,
+                            0.0008915319922380149,
+                            0.0009042309829965234,
+                            0.0010161853861063719,
+                            0.0009064966579899192,
+                            0.0007977227214723825,
+                            0.000862289743963629,
+                            0.0009978791931644082,
+                            0.0014485543360933661,
+                            0.0008234521956183016,
+                            0.00123439054004848,
+                            0.0009559749159961939,
+                            0.0009517490398138762,
+                            0.001359558547846973,
+                            0.0008559443522244692,
+                            0.0009189079282805324,
+                            0.001149592804722488,
+                            0.0009482887689955533,
+                            0.0010015141451731324,
+                            0.0014367529656738043,
+                            0.0013949282001703978,
+                            0.0008478481322526932,
+                            0.0009486040798947215,
+                            0.0008063078857958317,
+                            0.0007884667138569057,
+                            0.000955446099396795,
+                            0.0010544518008828163,
+                            0.0012754665222018957,
+                            0.0009307416621595621,
+                            0.0012456123949959874,
+                            0.0011768859112635255,
+                            0.0009335617651231587,
+                            0.0006628126720897853,
+                            0.0010911811841651797,
+                            0.0012401333078742027,
+                            0.0009072108659893274,
+                            0.0007884680526331067,
+                            0.0008576667751185596,
+                            0.0010973347816616297,
+                            0.0009842119179666042,
+                            0.0008329907432198524,
+                            0.0010382354957982898,
+                            0.0008177062845788896,
+                            0.0011125939199700952,
+                            0.0008802484371699393,
+                            0.0007749152719043195,
+                            0.0012204849626868963,
+                            0.0009280128870159388,
+                            0.0008357075857929885,
+                            0.0013957697665318847,
+                            0.0008502030977979302,
+                            0.0009454236715100706,
+                            0.0011369240237399936,
+                            0.001038751332089305,
+                            0.0008620588341727853,
+                            0.0007298748823814094,
+                            0.000925341562833637,
+                            0.0008653716067783535,
+                            0.0008903368725441396,
+                            0.0012279199436306953,
+                            0.000845591479446739,
+                            0.000819023116491735,
+                            0.0011881149839609861,
+                            0.0011747251264750957,
+                            0.0013162961695343256,
+                            0.000738493341486901,
+                            0.001226521097123623,
+                            0.0011371171567589045,
+                            0.0011164264287799597,
+                            0.0010585170239210129,
+                            0.0011822179658338428,
+                            0.0009397975518368185,
+                            0.0013559580547735095,
+                            0.001518893986940384,
+                            0.000919750367756933,
+                            0.001130785676650703,
+                            0.0009862129809334874,
+                            0.0011250582756474614,
+                            0.0009319133823737502,
+                            0.0011567504843696952,
+                            0.0010494499001652002,
+                            0.0012359604006633162,
+                            0.0007846808293834329,
+                            0.0011824643006548285,
+                            0.0010962928645312786,
+                            0.0010061983484774828,
+                            0.0008717612945474684,
+                            0.001138012041337788,
+                            0.0009285348933190107,
+                            0.0008491899352520704,
+                            0.0007641129777766764,
+                            0.0012658343184739351,
+                            0.0008501861011609435,
+                            0.0011195631232112646,
+                            0.0006927368813194335,
+                            0.0008600314031355083,
+                            0.0010934454621747136,
+                            0.0011110826162621379,
+                            0.0007978686480782926,
+                            0.0009212110890075564,
+                            0.0007194632780738175,
+                            0.000960988225415349,
+                            0.0008381536463275552,
+                            0.0010877682361751795,
+                            0.0009395431261509657,
+                            0.0011141253635287285,
+                            0.0017983298748731613,
+                            0.0007869705441407859,
+                            0.0017573647201061249,
+                            0.0013080001808702946,
+                            0.0009246825939044356,
+                            0.0008576754480600357,
+                            0.001091581187210977,
+                            0.0009753324557095766,
+                            0.0009317593649029732,
+                            0.00119885359890759,
+                            0.0017776066670194268,
+                            0.0008839459624141455,
+                            0.0009238495258614421,
+                            0.0007534348987974226,
+                            0.0011970774503424764,
+                            0.0014117001555860043,
+                            0.0013048017863184214,
+                            0.0009518149308860302,
+                            0.0011358908377587795,
+                            0.00089292909251526,
+                            0.0008536982350051403,
+                            0.000999008072540164,
+                            0.0007846413645893335,
+                            0.000838868785649538,
+                            0.0012880595168098807,
+                            0.000844885129481554,
+                            0.0009536961442790926,
+                            0.0008715416770428419,
+                            0.0008562793955206871,
+                            0.0007589749293401837,
+                            0.0012146608205512166,
+                            0.0011971329804509878,
+                            0.0010104571701958776,
+                            0.001094223465770483,
+                            0.0009234891622327268,
+                            0.000975425005890429,
+                            0.0010812339605763555,
+                            0.001356402412056923,
+                            0.0012176454765722156,
+                            0.0008644547197036445,
+                            0.0010729467030614614,
+                            0.0008890077006071806,
+                            0.0010383307235315442,
+                            0.0011185761541128159,
+                            0.0009039329015649855,
+                            0.0011175607796758413,
+                            0.0010541980154812336,
+                            0.00114694784861058,
+                            0.0013067093677818775,
+                            0.0010522144148126245,
+                            0.0009111643303185701,
+                            0.0008480974356643856,
+                            0.0012618761975318193,
+                            0.0008401052327826619,
+                            0.0012542551849037409,
+                            0.0008523935684934258,
+                            0.0013027016539126635,
+                            0.0009044892503879964,
+                            0.0009706730488687754,
+                            0.0009544889908283949,
+                            0.001190773444250226,
+                            0.0011414955370128155,
+                            0.0008576958789490163,
+                            0.0012597921304404736,
+                            0.0008738233591429889,
+                            0.0007626846199855208,
+                            0.000946465355809778,
+                            0.0011027485597878695,
+                            0.0009712046012282372,
+                            0.001269070548005402,
+                            0.0010537098860368133,
+                            0.0009414184605702758,
+                            0.001068871933966875,
+                            0.001339591690339148,
+                            0.0011699150782078505,
+                            0.0012016132241114974,
+                            0.0008075058576650918,
+                            0.0009043558384291828,
+                            0.0006492871325463057,
+                            0.0009367606835439801,
+                            0.001871450454927981,
+                            0.0009079722804017365,
+                            0.0008040340035222471,
+                            0.0011473165359348059,
+                            0.0009472420788370073,
+                            0.001051472150720656,
+                            0.0009369811159558594,
+                            0.0011982975993305445
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 68,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.014896157197654247
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "7": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "inter_arch.W_linears.2",
+                "input_tensor_infos": [
+                    {
+                        "id": 68,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.014896157197654247
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0009873901726678014,
+                            0.0008334061712957919,
+                            0.0006161424680612981,
+                            0.0009596861782483757,
+                            0.0009329541935585439,
+                            0.0006046370835974813,
+                            0.0007242577848955989,
+                            0.0006409888155758381,
+                            0.0005692478152923286,
+                            0.0014434970216825604,
+                            0.0005284189246594906,
+                            0.0009128038655035198,
+                            0.0005150813376531005,
+                            0.0005642138421535492,
+                            0.0006532635306939483,
+                            0.0006281661335378885,
+                            0.0006649557035416365,
+                            0.0009152569691650569,
+                            0.0009506570058874786,
+                            0.0005159039283171296,
+                            0.0006743535050190985,
+                            0.0006000508437864482,
+                            0.0005889972089789808,
+                            0.0005732599529437721,
+                            0.000564886606298387,
+                            0.0005316314636729658,
+                            0.00085851812036708,
+                            0.0005717577296309173,
+                            0.0007152939797379076,
+                            0.0005666211945936084,
+                            0.0006622542859986424,
+                            0.0008872043690644205,
+                            0.0006758521776646376,
+                            0.0005776151665486395,
+                            0.0006003616726957262,
+                            0.0005980476853437722,
+                            0.0005436306237243116,
+                            0.000587401504162699,
+                            0.0005935709923505783,
+                            0.0005767319817095995,
+                            0.0006185212405398488,
+                            0.0013778175925835967,
+                            0.0010117583442479372,
+                            0.000614158168900758,
+                            0.0006044075707904994,
+                            0.001092814258299768,
+                            0.0006134991417638958,
+                            0.0006626335671171546,
+                            0.0011323349317535758,
+                            0.0010283916490152478,
+                            0.0006998368189670146,
+                            0.0006455219117924571,
+                            0.0006468931096605957,
+                            0.0011226674541831017,
+                            0.0005414857878349721,
+                            0.0005274316063150764,
+                            0.0005130344070494175,
+                            0.0004591039614751935,
+                            0.0010638672392815351,
+                            0.0006035440601408482,
+                            0.0006608894327655435,
+                            0.0010218987008556724,
+                            0.0005662933108396828,
+                            0.0005579803255386651,
+                            0.0008229039376601577,
+                            0.0005150067736394703,
+                            0.0006154504953883588,
+                            0.0012511966051533818,
+                            0.0008195549598895013,
+                            0.0005840587546117604,
+                            0.0006265994161367416,
+                            0.0006014781538397074,
+                            0.0011359124910086393,
+                            0.0005668317317031324,
+                            0.0008401244995184243,
+                            0.0005880498210899532,
+                            0.0005572999943979084,
+                            0.0005941270501352847,
+                            0.0005670364480465651,
+                            0.0005270603578537703,
+                            0.0005773944430984557,
+                            0.0006995498551987112,
+                            0.0005490640760399401,
+                            0.0005367544363252819,
+                            0.0004978698561899364,
+                            0.000601290026679635,
+                            0.0005804267711937428,
+                            0.0011270653922110796,
+                            0.0012010468635708094,
+                            0.0006295142229646444,
+                            0.0005467869923450053,
+                            0.0005328480619937181,
+                            0.0007093461463227868,
+                            0.0005658024456351995,
+                            0.001043385942466557,
+                            0.0005630752420984209,
+                            0.0009546898072585464,
+                            0.0006423649028874934,
+                            0.0005543286679312587,
+                            0.0011854084441438317,
+                            0.0005191945238038898,
+                            0.0007516759214922786,
+                            0.0005968517507426441,
+                            0.0004957245546393096,
+                            0.000538662716280669,
+                            0.0007490225252695382,
+                            0.0005204853368923068,
+                            0.0005759045016020536,
+                            0.0010524654062464833,
+                            0.0005249017849564552,
+                            0.0005457442020997405,
+                            0.000565958849620074,
+                            0.0009732667240314186,
+                            0.0005657539004459977,
+                            0.0005729198455810547,
+                            0.0006477001006715,
+                            0.0005778660415671766,
+                            0.0010131701128557324,
+                            0.0005804079119116068,
+                            0.0012582105118781328,
+                            0.0005702163907699287,
+                            0.0005943109281361103,
+                            0.0006661994848400354,
+                            0.0006166524835862219,
+                            0.0005343815428204834,
+                            0.0006332485936582088,
+                            0.0006084199412725866,
+                            0.0006093116826377809,
+                            0.0007820291211828589,
+                            0.0006500287563540041,
+                            0.0005794161115773022,
+                            0.0005982551956549287,
+                            0.0006156038725748658,
+                            0.0007763697649352252,
+                            0.0006016955594532192,
+                            0.0006482041790150106,
+                            0.000578514183871448,
+                            0.0006362019921652973,
+                            0.000610069022513926,
+                            0.0005947016761638224,
+                            0.0006852048099972308,
+                            0.0005919733666814864,
+                            0.0006276134517975152,
+                            0.000555060338228941,
+                            0.0006181145436130464,
+                            0.000575713231228292,
+                            0.0005849584122188389,
+                            0.0006493897526524961,
+                            0.0007411661208607256,
+                            0.0006995082949288189,
+                            0.0005045281141065061,
+                            0.0006543661002069712,
+                            0.0007020715274848044,
+                            0.0006530442624352872,
+                            0.0007776790880598128,
+                            0.0005702633643522859,
+                            0.0005161202279850841,
+                            0.0005262107588350773,
+                            0.0006084329215809703,
+                            0.0006363837164826691,
+                            0.0007642265991307795,
+                            0.0005629016668535769,
+                            0.0005672003026120365,
+                            0.0005241086473688483,
+                            0.0006634830497205257,
+                            0.0008040121756494045,
+                            0.0009280675440095365,
+                            0.0005341465002857149,
+                            0.000649584922939539,
+                            0.0005665223579853773,
+                            0.0008710403926670551,
+                            0.0006404545274563134,
+                            0.0008244572090916336,
+                            0.00051411910681054,
+                            0.0005442971014417708,
+                            0.0005883976118639112,
+                            0.0006157684838399291,
+                            0.0006342388805933297,
+                            0.0005076139350421727,
+                            0.0006709498120471835,
+                            0.000683859339915216,
+                            0.0005323135992512107,
+                            0.0006029805517755449,
+                            0.0006375901866704226,
+                            0.0006982138729654253,
+                            0.0005245814099907875,
+                            0.0007060064235702157,
+                            0.0006015279795974493,
+                            0.0005541498539969325,
+                            0.000669327040668577,
+                            0.0005806542467325926,
+                            0.0007424579816870391,
+                            0.000641089805867523,
+                            0.0006255326443351805,
+                            0.0006204040255397558,
+                            0.0006036806735210121,
+                            0.0007168743759393692,
+                            0.0006610429263673723,
+                            0.0006031134980730712,
+                            0.0006309723830781877,
+                            0.0005738175823353231,
+                            0.0007726041949354112,
+                            0.0007854551658965647,
+                            0.0006625191890634596,
+                            0.0005940133123658597,
+                            0.000588780443649739,
+                            0.0005446261493489146,
+                            0.0007025277009233832,
+                            0.0007406820077449083,
+                            0.0006090003880672157,
+                            0.0006336247315630317,
+                            0.0007304392056539655,
+                            0.0005638569127768278,
+                            0.0004951116861775517,
+                            0.0005737284664064646,
+                            0.0005954639054834843,
+                            0.000571277632843703,
+                            0.0006312244804576039,
+                            0.000533736718352884,
+                            0.0005438094376586378,
+                            0.0007199951214715838,
+                            0.0005839496734552085,
+                            0.0007183169946074486,
+                            0.0005926074809394777,
+                            0.0005719332839362323,
+                            0.0006615512538701296,
+                            0.0006772509077563882,
+                            0.0005532169016078115,
+                            0.0006196595495566726,
+                            0.0005283661303110421,
+                            0.0006399123230949044,
+                            0.0007887117681093514,
+                            0.0005603019963018596,
+                            0.0008967852336354554,
+                            0.0008104944135993719,
+                            0.0005930213956162333,
+                            0.0005674827843904495,
+                            0.0006434547831304371,
+                            0.0005463427514769137,
+                            0.0007071365835145116,
+                            0.0006243860116228461,
+                            0.0006384140579029918,
+                            0.0005977627006359398,
+                            0.0005369451246224344,
+                            0.0007369007798843086,
+                            0.0005781689542345703,
+                            0.0006752123590558767,
+                            0.0005318341427482665,
+                            0.0007240258855745196,
+                            0.0007046410464681685,
+                            0.0007445156807079911,
+                            0.0007127068820409477,
+                            0.0005685148644261062,
+                            0.0007563360268250108,
+                            0.0007029222906567156,
+                            0.0005370762664824724,
+                            0.0007605223800055683,
+                            0.000582126434892416,
+                            0.0005226742941886187,
+                            0.0008174525573849678,
+                            0.0006111246766522527,
+                            0.0006296970532275736,
+                            0.0006599055486731231,
+                            0.0006300928653217852,
+                            0.0007700762944296002,
+                            0.000614394259173423,
+                            0.0005216817953623831,
+                            0.0007134860497899354,
+                            0.0008994771051220596,
+                            0.0007483653025701642,
+                            0.0005835272022522986,
+                            0.0007452891441062093,
+                            0.0007043418590910733,
+                            0.0006148119573481381,
+                            0.001082361675798893,
+                            0.0009179330081678927,
+                            0.0005647264770232141,
+                            0.0006136739393696189,
+                            0.0006531851831823587,
+                            0.0005778342019766569,
+                            0.0009267586283385754,
+                            0.0007616417715325952,
+                            0.0006505749770440161,
+                            0.000734490982722491,
+                            0.000570188625715673,
+                            0.0007504669483751059,
+                            0.000656279269605875,
+                            0.000680993776768446,
+                            0.0004971410380676389,
+                            0.0006252874154597521,
+                            0.0008397242054343224,
+                            0.0006944075576029718,
+                            0.0005020152311772108,
+                            0.0006176559254527092,
+                            0.0007496938342228532,
+                            0.0006627885741181672,
+                            0.0005570028442889452,
+                            0.0006728569278493524,
+                            0.000581247906666249,
+                            0.000623911852017045,
+                            0.0006298014777712524,
+                            0.0006226527038961649,
+                            0.0008409068104811013,
+                            0.0007690992788411677,
+                            0.0007110602455213666,
+                            0.0009108477970585227,
+                            0.0005767807015217841,
+                            0.000715885020326823,
+                            0.0008272331324405968,
+                            0.0009650561260059476,
+                            0.0007184746791608632,
+                            0.0005128328339196742,
+                            0.0006899447762407362,
+                            0.0008376571349799633,
+                            0.000649947440251708,
+                            0.0007851901464164257,
+                            0.0008545364253222942,
+                            0.0006878605345264077,
+                            0.0007013127324171364,
+                            0.0005404454423114657,
+                            0.0007634460926055908,
+                            0.0007618163363076746,
+                            0.0006475986447185278,
+                            0.0006195262540131807,
+                            0.000682723883073777,
+                            0.0005252982955425978,
+                            0.0005451248725876212,
+                            0.0007894642767496407,
+                            0.0005952318897470832,
+                            0.0007790143718011677,
+                            0.0006351543706841767,
+                            0.0006084928754717112,
+                            0.0007106178672984242,
+                            0.0005511320778168738,
+                            0.0007208827300928533,
+                            0.0007120047812350094,
+                            0.0009020795114338398,
+                            0.0005907882004976273,
+                            0.0006508937804028392,
+                            0.000641247199382633,
+                            0.0005770963616669178,
+                            0.0009753708145581186,
+                            0.0007857413729652762,
+                            0.0006783694843761623,
+                            0.0007090072031132877,
+                            0.0010393941774964333,
+                            0.0009272082825191319,
+                            0.0008887680014595389,
+                            0.0009139754110947251,
+                            0.0006980275502428412,
+                            0.0006475130212493241,
+                            0.0008299719193018973,
+                            0.0005271948175504804,
+                            0.0005259087774902582,
+                            0.0006765224388800561,
+                            0.000673743081279099,
+                            0.0008814479224383831,
+                            0.0008286409429274499,
+                            0.0005680841277353466,
+                            0.001069989986717701,
+                            0.0005959020345471799,
+                            0.0005758642801083624,
+                            0.0005899547832086682,
+                            0.0005981878493912518,
+                            0.000680457626003772,
+                            0.0007527790730819106,
+                            0.0005942084826529026,
+                            0.000578617793507874,
+                            0.000877470476552844,
+                            0.0007558297947980464,
+                            0.0008732116548344493,
+                            0.000737140653654933,
+                            0.000603396212682128,
+                            0.0006167832179926336,
+                            0.0008266773074865341,
+                            0.0005501699633896351,
+                            0.0009237322374247015,
+                            0.0006344463326968253,
+                            0.000629653106443584,
+                            0.0004987543215975165,
+                            0.000985778751783073,
+                            0.000809510238468647,
+                            0.0008942470885813236,
+                            0.000812788843177259,
+                            0.0006095516146160662,
+                            0.001026746118441224,
+                            0.0010048355907201767,
+                            0.0007936639012768865,
+                            0.0006766277365386486,
+                            0.0006338648963719606,
+                            0.0005213043186813593,
+                            0.0005736307357437909,
+                            0.0011619804427027702,
+                            0.0007367883808910847,
+                            0.0009558117599226534,
+                            0.0006244548712857068,
+                            0.000715897826012224,
+                            0.0008631824166513979,
+                            0.0005656270077452064,
+                            0.000902706291526556,
+                            0.0007987748831510544,
+                            0.0005795415490865707,
+                            0.000608309346716851,
+                            0.0007072545704431832,
+                            0.0007582537364214659,
+                            0.000953293580096215,
+                            0.0008168191998265684,
+                            0.0006225951365195215,
+                            0.000820311950519681,
+                            0.0007916140020824969,
+                            0.0007867238018661737,
+                            0.0008758484618738294,
+                            0.0008184629841707647,
+                            0.0007992333266884089,
+                            0.0008096659439615905,
+                            0.0007258905097842216,
+                            0.000578019127715379,
+                            0.0008856342174112797,
+                            0.0005994818056933582,
+                            0.0011798071209341288,
+                            0.0010364766931161284,
+                            0.000695264374371618,
+                            0.0008586780168116093,
+                            0.0006805704906582832,
+                            0.0006406469619832933,
+                            0.0007928204140625894,
+                            0.0010383184999227524,
+                            0.0005958741530776024,
+                            0.0006708441069349647,
+                            0.0006510422099381685,
+                            0.0009234647150151432,
+                            0.0007863118080422282,
+                            0.0010117222554981709,
+                            0.0006899168947711587,
+                            0.0008080674451775849,
+                            0.0007653621723875403,
+                            0.000732562446501106,
+                            0.0006759678362868726,
+                            0.0007295457180589437,
+                            0.0007094481261447072,
+                            0.0006617069011554122,
+                            0.0007010205881670117,
+                            0.0009056888520717621,
+                            0.0009467246127314866,
+                            0.0013817986473441124,
+                            0.0008987868204712868,
+                            0.000651539012324065,
+                            0.0009018174605444074,
+                            0.0007088822894729674,
+                            0.0005259619210846722,
+                            0.0007050696876831353,
+                            0.0009371790220029652,
+                            0.0007394482963718474,
+                            0.0009467842755839229,
+                            0.0005610868101939559,
+                            0.0006231462466530502,
+                            0.0007694741943851113,
+                            0.0008072687196545303,
+                            0.0006450553773902357,
+                            0.0013250557240098715,
+                            0.0007470045820809901,
+                            0.0007277231197804213,
+                            0.0007617620285600424,
+                            0.0006775339134037495,
+                            0.0007094931206665933,
+                            0.000933359086047858,
+                            0.0010630084434524179,
+                            0.0006856221007183194,
+                            0.0007873533759266138,
+                            0.0006574875442311168,
+                            0.0006127734668552876,
+                            0.0005744335940107703,
+                            0.0007553781033493578,
+                            0.0007937365444377065,
+                            0.0006010259967297316,
+                            0.0008297213935293257,
+                            0.0008386675035580993,
+                            0.0005017051007598639,
+                            0.0006817275425419211,
+                            0.0006832040380686522,
+                            0.0006229989230632782,
+                            0.0005988942575640976,
+                            0.0006126351072452962,
+                            0.0006277681677602232,
+                            0.0009064930491149426,
+                            0.0007134248153306544,
+                            0.0007750656805001199,
+                            0.0007679283153265715,
+                            0.0005361965158954263,
+                            0.0006453876267187297,
+                            0.0006039139698259532,
+                            0.0007547038258053362,
+                            0.0005734499427489936,
+                            0.0007187933661043644,
+                            0.0007079820497892797,
+                            0.0005688845994882286,
+                            0.0007308386848308146,
+                            0.0005540439742617309,
+                            0.0007881146157160401,
+                            0.0009868806228041649,
+                            0.000881162122823298,
+                            0.000584557477850467,
+                            0.00112270493991673,
+                            0.0006851215730421245,
+                            0.0011257630540058017,
+                            0.0006563685019500554,
+                            0.001103821094147861,
+                            0.0006904270267114043,
+                            0.0009639926138333976,
+                            0.0008205187623389065,
+                            0.0007790726376697421,
+                            0.0005985453608445823,
+                            0.0006167663959786296,
+                            0.0006051533855497837,
+                            0.0005835522315464914,
+                            0.0006100598257035017,
+                            0.0006969601381570101,
+                            0.0005782897933386266,
+                            0.0007685049786232412,
+                            0.0005339536000974476,
+                            0.0006104321801103652,
+                            0.00055299251107499,
+                            0.0005140176508575678,
+                            0.0005483813001774251,
+                            0.0005573403323069215,
+                            0.0007110349251888692,
+                            0.000555917271412909,
+                            0.0006427331827580929,
+                            0.0005331330467015505,
+                            0.0005021458491683006,
+                            0.0006810477934777737,
+                            0.0005776520119979978,
+                            0.0006072259857319295,
+                            0.0005929335602559149,
+                            0.000737807946279645,
+                            0.0006736816721968353,
+                            0.0005514785298146307,
+                            0.0005858078366145492,
+                            0.0005502867279574275,
+                            0.000666907406412065,
+                            0.0006039738073013723,
+                            0.0006467701168730855,
+                            0.0006583486683666706,
+                            0.0005769012495875359,
+                            0.0005469557945616543,
+                            0.0005603826721198857,
+                            0.0006944579654373229,
+                            0.0005783831584267318,
+                            0.0006124456995166838,
+                            0.0005130300996825099,
+                            0.0006084098713472486,
+                            0.0006117344019003212,
+                            0.0005509299808181822,
+                            0.0005774449673481286,
+                            0.0005653698462992907,
+                            0.0005547284381464124,
+                            0.0006278255605138838,
+                            0.0005113635561428964,
+                            0.0005429632146842778,
+                            0.0005302570061758161,
+                            0.0005113267106935382,
+                            0.0006606898969039321,
+                            0.0005669635138474405,
+                            0.0006561939371749759,
+                            0.0005659810267388821,
+                            0.0005052556516602635,
+                            0.0006433897069655359,
+                            0.0007226895540952682,
+                            0.0006433457601815462,
+                            0.0005153834936209023,
+                            0.0005927879828959703,
+                            0.0005080534610897303,
+                            0.0004980530356988311,
+                            0.0005553796072490513,
+                            0.0005072892527095973,
+                            0.0005423922557383776,
+                            0.0005681168986484408,
+                            0.0006606280221603811,
+                            0.0004914705059491098,
+                            0.0006752873305231333,
+                            0.0006868473719805479,
+                            0.0006447097985073924,
+                            0.0005725871305912733,
+                            0.0005610843072645366,
+                            0.0005160868167877197,
+                            0.000815408886410296,
+                            0.000540647772140801,
+                            0.0005495942896232009,
+                            0.000595740566495806,
+                            0.0004859508480876684,
+                            0.0005734097212553024,
+                            0.0005344473756849766,
+                            0.0005869986489415169,
+                            0.0006571486010216177,
+                            0.0006094187265262008,
+                            0.0006208374397829175,
+                            0.0006569188553839922,
+                            0.0005788560956716537,
+                            0.000670391193125397,
+                            0.0005899074021726847,
+                            0.0006157398456707597,
+                            0.0005515041993930936,
+                            0.0005840620142407715,
+                            0.0005646029021590948,
+                            0.0006485985359176993,
+                            0.000635531498119235,
+                            0.0006578320171684027,
+                            0.0006628408445976675,
+                            0.000525629729963839,
+                            0.0005736681632697582,
+                            0.0005290688131935894,
+                            0.0005605207988992333,
+                            0.0005038367235101759,
+                            0.0007056788890622556,
+                            0.0005515259690582752,
+                            0.0005690849502570927,
+                            0.0006449228967539966,
+                            0.0006542913615703583,
+                            0.000569443975109607,
+                            0.0004893661243841052,
+                            0.0005876807845197618,
+                            0.0006432380178011954,
+                            0.0006584167131222785,
+                            0.0005816597258672118,
+                            0.0005614050314761698,
+                            0.0004996141651645303,
+                            0.0006045220070518553,
+                            0.0005774881574325264,
+                            0.0006656672921963036,
+                            0.0005538715631701052,
+                            0.0005452685290947556,
+                            0.0005960890557616949,
+                            0.0005805836990475655,
+                            0.0004883886431343853,
+                            0.0006181541830301285,
+                            0.000699525757227093,
+                            0.0005017883377149701,
+                            0.0005608917563222349,
+                            0.0006058808648958802,
+                            0.0005796538898721337,
+                            0.0005737805040553212,
+                            0.0005262354388833046,
+                            0.0005148454802110791,
+                            0.00048645143397152424,
+                            0.0005716747837141156,
+                            0.0006662655505351722,
+                            0.0005931745981797576,
+                            0.0005781055078841746,
+                            0.0006292793550528586,
+                            0.0005587316118180752,
+                            0.0005466660368256271,
+                            0.0006391559145413339,
+                            0.0005123070441186428,
+                            0.0004701846628449857,
+                            0.0005538482801057398,
+                            0.0005997717962600291,
+                            0.0005673955893144011,
+                            0.0005946033634245396,
+                            0.0005704625509679317,
+                            0.0005224469350650907,
+                            0.0005140145076438785,
+                            0.0006218370981514454,
+                            0.0006178991170600057,
+                            0.0005183597095310688,
+                            0.0006122784689068794,
+                            0.0005075552617199719,
+                            0.0004935793695040047,
+                            0.0006372594507411122,
+                            0.0005252109258435667,
+                            0.0005666537908837199,
+                            0.0006190456915646791,
+                            0.0005866518476977944,
+                            0.0004978175275027752,
+                            0.000659393728710711,
+                            0.0004762189637403935,
+                            0.0005853252368979156,
+                            0.000588641909416765,
+                            0.0005965417949482799,
+                            0.000582316773943603,
+                            0.0006131519330665469,
+                            0.0006624925881624222,
+                            0.0006505285273306072,
+                            0.0005457258084788918,
+                            0.0005644637858495116,
+                            0.0005499997059814632,
+                            0.0005298529285937548,
+                            0.0005005284328944981,
+                            0.0006577749154530466,
+                            0.0005834513576701283,
+                            0.0005955290398560464,
+                            0.0006809931946918368,
+                            0.000491869286634028,
+                            0.0004808821249753237,
+                            0.0006042449967935681,
+                            0.0005618453142233193,
+                            0.0005257211159914732,
+                            0.000543939764611423,
+                            0.000594055512920022,
+                            0.0006107547087594867,
+                            0.0005561219877563417,
+                            0.0005918772658333182,
+                            0.0006781129632145166,
+                            0.0006558353779837489,
+                            0.000600246072281152,
+                            0.000541097077075392,
+                            0.0006171250133775175,
+                            0.0006184782250784338,
+                            0.0005972498911432922,
+                            0.0007965034455992281,
+                            0.0006385665037669241,
+                            0.0006557301967404783,
+                            0.0005633734399452806,
+                            0.0006076812278479338,
+                            0.0005187254282645881,
+                            0.000506781623698771,
+                            0.0005526978638954461,
+                            0.0006844606250524521,
+                            0.0005077518871985376,
+                            0.000601272564381361,
+                            0.0005568811320699751,
+                            0.0007215534569695592,
+                            0.0005570456269197166,
+                            0.0004892470897175372,
+                            0.0005764301167801023,
+                            0.0005097895045764744,
+                            0.0005746035021729767,
+                            0.0005459937383420765,
+                            0.0005634939298033714,
+                            0.0006287164869718254,
+                            0.0006496130372397602,
+                            0.0005907292943447828,
+                            0.0005312769208103418,
+                            0.0005533731309697032,
+                            0.000604950007982552,
+                            0.0006524731288664043,
+                            0.0005379404174163938,
+                            0.0006847081822343171,
+                            0.0005455799400806427,
+                            0.000600817205850035,
+                            0.0005791831063106656,
+                            0.0005881682736799121,
+                            0.0005679124151356518,
+                            0.0006287937867455184,
+                            0.0005431415629573166,
+                            0.0006621320499107242,
+                            0.0005782430525869131,
+                            0.0005626219208352268,
+                            0.0005154248210601509,
+                            0.0005802956875413656,
+                            0.0006192216533236206,
+                            0.0005172239034436643,
+                            0.0005925072473473847,
+                            0.0006873452803120017,
+                            0.0005075633525848389,
+                            0.0005462178960442543,
+                            0.0005354147870093584,
+                            0.0006024311296641827,
+                            0.0004487800470087677,
+                            0.000557522929739207,
+                            0.0005600509466603398,
+                            0.0005605370970442891,
+                            0.0005190914962440729,
+                            0.0005260682082735002,
+                            0.0005128514603711665,
+                            0.0006865847972221673,
+                            0.0005186879425309598,
+                            0.000594373675994575,
+                            0.000646308995783329,
+                            0.0005746406968683004,
+                            0.0005225770291872323,
+                            0.0005505462759174407,
+                            0.0005406573764048517,
+                            0.0005462259869091213,
+                            0.0006876559928059578,
+                            0.0005695936852134764,
+                            0.0007445791270583868,
+                            0.0005257108132354915,
+                            0.0004778190341312438,
+                            0.0005872643669135869,
+                            0.000541489280294627,
+                            0.000524071860127151,
+                            0.0006286021671257913,
+                            0.0006249622674658895,
+                            0.0005496906233020127,
+                            0.0005075062508694828,
+                            0.0006362689891830087,
+                            0.0005517933168448508,
+                            0.0005898501258343458,
+                            0.0005520711420103908,
+                            0.0005179299623705447,
+                            0.0006380291888490319,
+                            0.0006830492056906223,
+                            0.0006600204505957663,
+                            0.0005921718548052013,
+                            0.0005172395030967891,
+                            0.0005731812561862171,
+                            0.0006565920193679631,
+                            0.0005966013995930552,
+                            0.000717372924555093,
+                            0.000700357195455581,
+                            0.0005046962760388851,
+                            0.0006527554942294955,
+                            0.000704644771758467,
+                            0.0006213899469003081,
+                            0.0005494570359587669,
+                            0.0005395300104282796,
+                            0.0005822249804623425,
+                            0.0006212058360688388,
+                            0.0006222000229172409,
+                            0.000590766139794141,
+                            0.0006164406077004969,
+                            0.0007314858376048505,
+                            0.000521636800840497,
+                            0.0005926404846832156,
+                            0.0005481512635014951,
+                            0.0006281799869611859,
+                            0.0005193882971070707,
+                            0.0006840992136858404,
+                            0.0005004940321668983,
+                            0.0006025951006449759,
+                            0.0006922520697116852,
+                            0.0005735572776757181,
+                            0.0005062977434135973,
+                            0.0005564458551816642,
+                            0.0005934872897341847,
+                            0.0005201013991609216,
+                            0.0005514904041774571,
+                            0.0007760904263705015,
+                            0.0005436044884845614,
+                            0.0005592055385932326,
+                            0.0006016354309394956,
+                            0.000611505180131644,
+                            0.0006209317361935973,
+                            0.0005368584534153342,
+                            0.0006218794733285904,
+                            0.0005528847104869783,
+                            0.0005136680556461215,
+                            0.0005585504113696516,
+                            0.0006989068351686001,
+                            0.000668927445076406,
+                            0.0006005714531056583,
+                            0.0007249080808833241,
+                            0.0005577749107033014,
+                            0.0005192381795495749,
+                            0.000664953317027539,
+                            0.0006303226691670716,
+                            0.0006153901922516525,
+                            0.0006672356394119561,
+                            0.0006041810265742242,
+                            0.0006128553650341928,
+                            0.0005805097753182054,
+                            0.000604256521910429,
+                            0.0005300520570017397,
+                            0.0006752129993401468,
+                            0.0006239557405933738,
+                            0.0006338975508697331,
+                            0.000549358082935214,
+                            0.0005341864307411015,
+                            0.0006426145555451512,
+                            0.0006009583012200892,
+                            0.0005806932458654046,
+                            0.0005578076234087348,
+                            0.0007638012175448239,
+                            0.0005022180848754942,
+                            0.0006883401074446738,
+                            0.0006824960000813007,
+                            0.0005242697661742568,
+                            0.0005735423183068633,
+                            0.0007310695364139974,
+                            0.0006336747319437563,
+                            0.0005379081703722477,
+                            0.0007160504464991391,
+                            0.0006668066489510238,
+                            0.0005049339961260557,
+                            0.0006068320362828672,
+                            0.0005599943688139319,
+                            0.0007848694804124534,
+                            0.0006562292692251503,
+                            0.000637975987046957,
+                            0.0006609109696000814,
+                            0.0005569913773797452,
+                            0.0005762559594586492,
+                            0.0008201127639040351,
+                            0.0006337834638543427,
+                            0.0005809200811199844,
+                            0.0005448693409562111,
+                            0.0006016995175741613,
+                            0.0006354852812364697,
+                            0.0005280484911054373,
+                            0.0005551778012886643,
+                            0.0005646617501042783,
+                            0.0005119456327520311,
+                            0.0005178931751288474,
+                            0.0005757675389759243,
+                            0.0005778790800832212,
+                            0.0005967218894511461,
+                            0.0010039606131613255,
+                            0.0005546581232920289,
+                            0.0009874033275991678,
+                            0.0009286241256631911,
+                            0.0006824773154221475,
+                            0.0005760205094702542,
+                            0.0006774698267690837,
+                            0.000559153500944376,
+                            0.0006947769434191287,
+                            0.0007092159939929843,
+                            0.0005736167076975107,
+                            0.0005803341628052294,
+                            0.000927782675717026,
+                            0.000681644247379154,
+                            0.000674027600325644,
+                            0.0008528346661478281,
+                            0.0005251922993920743,
+                            0.0007880457560531795,
+                            0.0006879015709273517,
+                            0.0010872415732592344,
+                            0.0006152333226054907,
+                            0.0007273866212926805,
+                            0.0007677973480895162,
+                            0.0008782726945355535,
+                            0.0006006982293911278,
+                            0.0007938036578707397,
+                            0.0008550865459255874,
+                            0.0007859999313950539,
+                            0.00122707593254745,
+                            0.0005839741788804531,
+                            0.0006435386021621525,
+                            0.0008180494769476354,
+                            0.0007692222134210169,
+                            0.0004863543435931206,
+                            0.0006076350691728294,
+                            0.000925281026866287,
+                            0.0008878648513928056,
+                            0.0007338461582548916,
+                            0.0013528125127777457,
+                            0.0009138846071437001,
+                            0.0010777815477922559,
+                            0.0006459845462813973,
+                            0.0005930314073339105,
+                            0.0006883606547489762,
+                            0.0005698407767340541,
+                            0.0007580089149996638,
+                            0.0005683650961145759,
+                            0.0009167385287582874,
+                            0.0008118361001834273,
+                            0.0009664861136116087,
+                            0.000574743200559169,
+                            0.0005840669036842883,
+                            0.0008850047015585005,
+                            0.0010712537914514542,
+                            0.0007836944423615932,
+                            0.0008425425621680915,
+                            0.0005396895576268435,
+                            0.000682351877912879,
+                            0.0012457112316042185,
+                            0.0007810599054209888,
+                            0.0006436947733163834,
+                            0.0007475895108655095,
+                            0.0007284091552719474,
+                            0.0006014729151502252,
+                            0.0009935032576322556,
+                            0.0007299623684957623,
+                            0.0006227538688108325,
+                            0.0006655335309915245,
+                            0.0006908627110533416,
+                            0.0009814376244321465,
+                            0.0007213514181785285,
+                            0.0005184650653973222,
+                            0.0006090537062846124,
+                            0.0006980188773013651,
+                            0.000608426402322948,
+                            0.0006512590334750712,
+                            0.0005637271679006517,
+                            0.0005199959268793464,
+                            0.0006371314520947635,
+                            0.0009671624866314232,
+                            0.0005397609784267843,
+                            0.0010200764518231153,
+                            0.0008234338602051139,
+                            0.0005755419842898846,
+                            0.0006754609639756382,
+                            0.0008044772548601031,
+                            0.0007266834145411849,
+                            0.0007063280208967626,
+                            0.0008979568374343216,
+                            0.0006527694640681148,
+                            0.0008125618915073574,
+                            0.0008306615054607391,
+                            0.000651308917440474,
+                            0.0004653239739127457,
+                            0.0006280806846916676,
+                            0.0004877620085608214,
+                            0.0007135575287975371,
+                            0.0006619059713557363,
+                            0.0005416636704467237,
+                            0.0007142163813114166,
+                            0.000864948786329478,
+                            0.0006318562664091587,
+                            0.0005887398147024214,
+                            0.0005759208579547703,
+                            0.0005726773524656892,
+                            0.0009434612584300339,
+                            0.0006540390313602984,
+                            0.0006346525042317808,
+                            0.0007477566250599921,
+                            0.0009183192159980536,
+                            0.0008970125345513225,
+                            0.0007238021353259683,
+                            0.000806479889433831,
+                            0.0008908644667826593,
+                            0.000740679504815489,
+                            0.0007002126076258719,
+                            0.0005425816052593291,
+                            0.000880659616086632,
+                            0.0008984377491287887,
+                            0.0005328375846147537,
+                            0.0008838206413201988,
+                            0.0006774616776965559,
+                            0.0006226715049706399,
+                            0.0009080951567739248,
+                            0.0009903489844873548,
+                            0.0006413959781639278,
+                            0.00068358622957021,
+                            0.0005578009295277297,
+                            0.000757149129640311,
+                            0.0005469544557854533,
+                            0.0005023248377256095,
+                            0.0007274351082742214,
+                            0.0006249711732380092,
+                            0.0006281205569393933,
+                            0.0005543911247514188,
+                            0.0006042653694748878,
+                            0.000558905943762511,
+                            0.0006183285731822252,
+                            0.000590358569752425,
+                            0.0005313197616487741,
+                            0.000741921947337687,
+                            0.0006682784878648818,
+                            0.0007098416681401432,
+                            0.0006578575703315437,
+                            0.0007494773017242551,
+                            0.000657068972941488,
+                            0.0006344660650938749,
+                            0.000584437686484307,
+                            0.0007049782434478402,
+                            0.000672730675432831,
+                            0.0006634776364080608,
+                            0.0007310188957490027,
+                            0.0005823670653626323,
+                            0.0006047696806490421,
+                            0.000586363603360951,
+                            0.0005602677119895816,
+                            0.0009762227418832481,
+                            0.0006447959458455443,
+                            0.0006397467222996056,
+                            0.0006097968434914947,
+                            0.0006521406467072666,
+                            0.0005922068376094103,
+                            0.0006113167037256062,
+                            0.0006535469437949359,
+                            0.0006354710785672069,
+                            0.0005804586107842624,
+                            0.000696757691912353,
+                            0.0005853985203430057,
+                            0.0005964372539892793,
+                            0.0005374624743126333,
+                            0.0006795093067921698,
+                            0.0006185791571624577,
+                            0.0006374369258992374,
+                            0.0007157765212468803,
+                            0.000562432745937258,
+                            0.0007592640467919409,
+                            0.0005874473717994988,
+                            0.0006465272163040936,
+                            0.0006197705515660346,
+                            0.0009277710923925042,
+                            0.0005462718545459211,
+                            0.0007229729671962559,
+                            0.0007733505917713046,
+                            0.0006426495965570211,
+                            0.0006013514357618988,
+                            0.0008827515412122011,
+                            0.0006090523675084114,
+                            0.0006443079910241067,
+                            0.0008528250036761165,
+                            0.0007315056282095611,
+                            0.0006663475069217384,
+                            0.0005951123894192278,
+                            0.0007343024481087923,
+                            0.0007515912293456495,
+                            0.0006210709689185023,
+                            0.0005264603532850742,
+                            0.00073172029806301,
+                            0.0005612727836705744,
+                            0.000656972115393728,
+                            0.0005806879489682615,
+                            0.000617949990555644,
+                            0.000748934515286237,
+                            0.0005927500897087157,
+                            0.000643206643871963,
+                            0.0006937221041880548,
+                            0.0005811292794533074,
+                            0.000944073311984539,
+                            0.0005775452591478825,
+                            0.0006858471897430718,
+                            0.0005802767118439078,
+                            0.0008627561619505286,
+                            0.0006596099119633436,
+                            0.0006747838924638927,
+                            0.0006574414437636733,
+                            0.0006362979183904827,
+                            0.0008401136146858335,
+                            0.0007434816216118634,
+                            0.0005879972595721483,
+                            0.0005948430043645203,
+                            0.0006459823343902826,
+                            0.0005036383518017828,
+                            0.0006229162099771202,
+                            0.0007140212110243738,
+                            0.0008256393484771252,
+                            0.0006750191678293049,
+                            0.0005616008420474827,
+                            0.0005311095155775547,
+                            0.0006053537945263088,
+                            0.0006819524569436908,
+                            0.0008132540970109403,
+                            0.0006021935841999948,
+                            0.0008046641014516354,
+                            0.0005976722459308803,
+                            0.0006391739589162171,
+                            0.0006222232477739453,
+                            0.0007918085320852697,
+                            0.0005920472904108465,
+                            0.0006724722334183753,
+                            0.0006437123520299792,
+                            0.000633154297247529,
+                            0.000598203216213733,
+                            0.0005978360422886908,
+                            0.0010508823907002807,
+                            0.0007036585593596101,
+                            0.0007729554199613631,
+                            0.0005336637841537595,
+                            0.0006622886867262423,
+                            0.0006355500663630664,
+                            0.0007247619214467704,
+                            0.0006091829272918403,
+                            0.0006457040435634553,
+                            0.000655800336971879,
+                            0.0006399978301487863,
+                            0.00048715114826336503,
+                            0.0006279678200371563,
+                            0.0006650342838838696,
+                            0.0006814011139795184,
+                            0.0005801177467219532,
+                            0.0005660034366883337,
+                            0.000573386496398598,
+                            0.0005538114346563816,
+                            0.0005600107251666486,
+                            0.0005760130588896573,
+                            0.0005534305819310248,
+                            0.0006842481670901179,
+                            0.0005942267598584294,
+                            0.0005651712999679148,
+                            0.0005131649086251855,
+                            0.0007561202510260046,
+                            0.0006434711394831538,
+                            0.000774143380112946,
+                            0.0011324952356517315,
+                            0.000633448304142803,
+                            0.000836336228530854,
+                            0.0007669358747079968,
+                            0.0006701217498630285,
+                            0.0005405893898569047,
+                            0.0006080454913899302,
+                            0.0006270998856052756,
+                            0.000803647912107408,
+                            0.0005141945439390838,
+                            0.000627222063485533,
+                            0.0005299351178109646,
+                            0.0006316581857390702,
+                            0.0006871946388855577,
+                            0.0007772863027639687,
+                            0.000584366382099688,
+                            0.0005512203206308186,
+                            0.0005685423966497183,
+                            0.0005873742629773915,
+                            0.0005828128778375685,
+                            0.0007875480805523694,
+                            0.0005768610863015056,
+                            0.000591342628467828,
+                            0.0007348464569076896,
+                            0.0005627249483950436,
+                            0.0005485753645189106,
+                            0.0006587171228602529,
+                            0.0007384582422673702,
+                            0.0005991474026814103,
+                            0.0006449103821069002,
+                            0.0005549205234274268,
+                            0.0005932511994615197,
+                            0.0005350942374207079,
+                            0.0006620236672461033,
+                            0.0005368960555642843,
+                            0.0006333487690426409,
+                            0.0006310597527772188,
+                            0.0007297493284568191,
+                            0.0005098633700981736,
+                            0.0004973332397639751,
+                            0.0005579510470852256,
+                            0.0005746001261286438,
+                            0.0007580269011668861,
+                            0.0007622361881658435,
+                            0.0005861058598384261,
+                            0.0006195965688675642,
+                            0.0005298652686178684,
+                            0.0006665182881988585,
+                            0.0006674189353361726,
+                            0.0006196401081979275,
+                            0.000830340723041445,
+                            0.0009021512232720852,
+                            0.0006196537869982421,
+                            0.0005720897461287677,
+                            0.0006238184287212789,
+                            0.0005258413148112595,
+                            0.0006724479608237743,
+                            0.0006417283439077437,
+                            0.0007530469447374344,
+                            0.0005450694006867707,
+                            0.0005169687210582197,
+                            0.0005660777678713202,
+                            0.0009099190356209874,
+                            0.0006064324406906962,
+                            0.0006959704332984984,
+                            0.000747468788176775,
+                            0.0005708329263143241,
+                            0.0005375633481889963,
+                            0.0006459722644649446,
+                            0.0006199392373673618,
+                            0.0006444878526963294,
+                            0.0005873155314475298,
+                            0.0006464057951234281,
+                            0.0006222726078704,
+                            0.0010629822500050068,
+                            0.0008174808463081717,
+                            0.0006354853394441307,
+                            0.0006494728731922805,
+                            0.0004535226908046752,
+                            0.0008083151187747717,
+                            0.0006306077702902257,
+                            0.0007319634314626455,
+                            0.0006361808627843857,
+                            0.0005899191019125283,
+                            0.0005921208066865802,
+                            0.0005946345045231283,
+                            0.0005256384029053152,
+                            0.0007662341231480241,
+                            0.000547020579688251,
+                            0.0005785755929537117,
+                            0.0005299903568811715,
+                            0.0006071932730264962,
+                            0.0005687669035978615,
+                            0.0006554043502546847,
+                            0.0005465737194754183,
+                            0.0006464160396717489,
+                            0.0006204625824466348,
+                            0.0007191414479166269,
+                            0.0006623429362662137,
+                            0.0005815731710754335,
+                            0.0006204611272551119,
+                            0.0005868584266863763,
+                            0.0006106479559093714,
+                            0.0005900042597204447,
+                            0.0006260345107875764,
+                            0.000846893759444356,
+                            0.0005392382154241204,
+                            0.0007007192471064627,
+                            0.000556137936655432,
+                            0.0005631546955555677,
+                            0.0005927685415372252,
+                            0.000552291632629931,
+                            0.0006361957639455795,
+                            0.00047283529420383275,
+                            0.0006828406476415694,
+                            0.0006411140784621239,
+                            0.0005194742698222399,
+                            0.0005902621196582913,
+                            0.0006182496435940266,
+                            0.0004789938684552908,
+                            0.0005918238894082606,
+                            0.0006448375061154366,
+                            0.0006442290614359081,
+                            0.0007226713350974023,
+                            0.0005852524191141129,
+                            0.0005944730364717543,
+                            0.0008041509427130222,
+                            0.0005889218882657588,
+                            0.0005096429958939552,
+                            0.0005584746249951422,
+                            0.000654602306894958,
+                            0.0006899972795508802,
+                            0.000702038814779371,
+                            0.000693308946210891,
+                            0.0005644102348014712,
+                            0.0005590140935964882,
+                            0.0006140089244581759,
+                            0.0005634254775941372,
+                            0.0007601903053000569,
+                            0.0007458231993950903,
+                            0.000643314968328923,
+                            0.0006929011433385313,
+                            0.0006400412530638278,
+                            0.0006727178115397692,
+                            0.0005978417466394603,
+                            0.000651883368846029,
+                            0.0006191327120177448,
+                            0.0005997156258672476,
+                            0.0007403839263133705,
+                            0.0008284313953481615,
+                            0.0005611132364720106,
+                            0.0006703572580590844,
+                            0.0006896809791214764,
+                            0.000586261332500726,
+                            0.0007959279464557767,
+                            0.0005521057173609734,
+                            0.000581784697715193,
+                            0.0007339465082623065,
+                            0.0005781208747066557,
+                            0.0006614128942601383,
+                            0.000656512042041868,
+                            0.000691067602019757,
+                            0.0005595553666353226,
+                            0.0006034295656718314,
+                            0.0005548486369661987,
+                            0.0005356612964533269,
+                            0.0006030547083355486,
+                            0.0005780381616204977,
+                            0.0007455368177033961,
+                            0.0006971788825467229,
+                            0.0005971328355371952,
+                            0.0005291577544994652,
+                            0.0006046710768714547,
+                            0.0005059749819338322,
+                            0.0007952948217280209,
+                            0.0008270525140687823,
+                            0.0006794980145059526,
+                            0.000539408007171005,
+                            0.0006537857698276639,
+                            0.0007334027905017138,
+                            0.0005761429201811552,
+                            0.0005911603220738471,
+                            0.0007243863074108958,
+                            0.0006783099961467087,
+                            0.0006508123478852212,
+                            0.0006397581310011446,
+                            0.000648513319902122,
+                            0.0007774544064886868,
+                            0.0006169128464534879,
+                            0.0005753351142629981,
+                            0.0006427274784073234,
+                            0.000554169062525034,
+                            0.0006943640182726085,
+                            0.0005389450816437602,
+                            0.0006484421319328249,
+                            0.0007062501390464604,
+                            0.0006751355831511319,
+                            0.0007131763850338757,
+                            0.0006313319900073111,
+                            0.0005872220499441028,
+                            0.0005807917914353311,
+                            0.0005506071029230952,
+                            0.0005845277337357402,
+                            0.0007932068547233939,
+                            0.0007010892149992287,
+                            0.0005924185970798135,
+                            0.0005992830265313387,
+                            0.0006355165969580412,
+                            0.0005602125311270356,
+                            0.0006322625558823347,
+                            0.0007410021498799324,
+                            0.0006076358840800822,
+                            0.0007418208988383412,
+                            0.0005941264680586755,
+                            0.0006140412297099829,
+                            0.0008242459734901786,
+                            0.0005567620974034071,
+                            0.0007180432439781725,
+                            0.0006470073130913079,
+                            0.0006010747165419161,
+                            0.0005082373390905559,
+                            0.0005658222362399101,
+                            0.0006738842930644751,
+                            0.0005582870217040181,
+                            0.0006347360904328525,
+                            0.0006638882914558053,
+                            0.0005974717787466943,
+                            0.0006330034811981022,
+                            0.0005396484048105776,
+                            0.00064150954131037,
+                            0.0005270905094221234,
+                            0.0006915373960509896,
+                            0.0005399166839197278,
+                            0.0006084433989599347,
+                            0.0005127057665959001,
+                            0.0006592794670723379,
+                            0.0006625557434745133,
+                            0.0006749997846782207,
+                            0.0006350876647047698,
+                            0.000546999042853713,
+                            0.0006648873095400631,
+                            0.0008656224817968905,
+                            0.0006896603154018521,
+                            0.0006908942596055567,
+                            0.0005491492920555174,
+                            0.0005183715256862342,
+                            0.0008306567324325442,
+                            0.000675289542414248,
+                            0.000551029690541327,
+                            0.000537309271749109,
+                            0.000614013581071049,
+                            0.0005420983070507646,
+                            0.0005280151381157339,
+                            0.0009302144753746688,
+                            0.0004899878986179829,
+                            0.0007469130214303732,
+                            0.0007586561841890216,
+                            0.0006347423768602312,
+                            0.000522528775036335,
+                            0.0007875910960137844,
+                            0.0005883777630515397,
+                            0.0006248790305107832,
+                            0.0005513433134183288,
+                            0.0005466192378662527,
+                            0.0006779201794415712,
+                            0.0006640529609285295,
+                            0.0006315490463748574,
+                            0.0006980120670050383,
+                            0.0007186898146755993,
+                            0.0006048515206202865,
+                            0.0006564107607118785,
+                            0.0006252544117160141,
+                            0.0006139952456578612,
+                            0.0006056365091353655,
+                            0.0005079874536022544,
+                            0.0005532082868739963,
+                            0.0005493318312801421,
+                            0.0008869640296325088,
+                            0.0007221201667562127,
+                            0.0006831969949416816,
+                            0.0007541250088252127,
+                            0.0007051155553199351,
+                            0.0006500966264866292,
+                            0.0006115639116615057,
+                            0.0007847128435969353,
+                            0.0006722628604620695,
+                            0.000640643120277673,
+                            0.0007845766958780587,
+                            0.0006276173517107964,
+                            0.0005127307376824319,
+                            0.0005385073600336909,
+                            0.0008670841343700886,
+                            0.000656325719319284,
+                            0.0008632162935100496,
+                            0.0005956863751634955,
+                            0.0007565299747511744,
+                            0.0006474708789028227,
+                            0.0006088798982091248,
+                            0.0007169343298301101,
+                            0.0006292227772064507,
+                            0.0007688375771977007,
+                            0.0006732019828632474,
+                            0.0006500845192931592,
+                            0.0006261882372200489,
+                            0.0005535283125936985,
+                            0.0005975653184577823,
+                            0.000570163712836802,
+                            0.0006744751008227468,
+                            0.0005246571381576359,
+                            0.0006462406599894166,
+                            0.0005806931294500828,
+                            0.000530431279912591,
+                            0.0008043410489335656,
+                            0.0007080035284161568,
+                            0.0006189412088133395,
+                            0.0006476885755546391,
+                            0.0006043523899279535,
+                            0.0005965649615973234,
+                            0.0007483719964511693,
+                            0.0006335288053378463,
+                            0.0006411722279153764,
+                            0.0006650813156738877,
+                            0.0005019113887101412,
+                            0.0005929591134190559,
+                            0.0005611997912637889,
+                            0.0011218406725674868,
+                            0.0005421548266895115,
+                            0.000582891923841089,
+                            0.0005840070080012083,
+                            0.0008167402702383697,
+                            0.0006279723020270467,
+                            0.0006970995455048978,
+                            0.0006434296956285834,
+                            0.00047636707313358784,
+                            0.0007065102690830827,
+                            0.0006430625217035413,
+                            0.0009134354186244309,
+                            0.0005478223320096731,
+                            0.0007295587565749884,
+                            0.0006159040494821966,
+                            0.000660163932479918,
+                            0.0005546833854168653,
+                            0.0006188569241203368,
+                            0.0007103094249032438,
+                            0.0005621624877676368,
+                            0.000519518565852195,
+                            0.0005422924878075719,
+                            0.0005517552490346134,
+                            0.000839708314742893,
+                            0.0005884796264581382,
+                            0.0007454906590282917,
+                            0.0006943001644685864,
+                            0.0006587171228602529,
+                            0.000683977676089853,
+                            0.000556122453417629,
+                            0.0006222018273547292,
+                            0.0007376598659902811,
+                            0.0007246423047035933,
+                            0.0004964524996466935,
+                            0.0006905860500410199,
+                            0.0005693674320355058,
+                            0.0005832196911796927,
+                            0.0006909819203428924,
+                            0.0005750468117184937,
+                            0.0007568610017187893,
+                            0.0007689483463764191,
+                            0.0005799726350232959,
+                            0.0005839407676830888,
+                            0.0006850327481515706,
+                            0.0006509643280878663,
+                            0.0006826944299973547,
+                            0.0007502196822315454,
+                            0.0005782665102742612,
+                            0.0005849126027897,
+                            0.000571995391510427,
+                            0.0007135606138035655,
+                            0.0006993486895225942,
+                            0.0006874110549688339,
+                            0.0006438207346946001,
+                            0.0006960854516364634,
+                            0.000564099638722837,
+                            0.0008518327376805246,
+                            0.0005529209156520665,
+                            0.0005536985117942095,
+                            0.0006869658245705068,
+                            0.0008628389914520085,
+                            0.0006562822964042425,
+                            0.0005530730122700334,
+                            0.0005771874566562474,
+                            0.0004912788281217217,
+                            0.0006668149144388735,
+                            0.0005405187839642167,
+                            0.0006311825709417462,
+                            0.0006535082357004285,
+                            0.0005857193609699607,
+                            0.0007450866978615522,
+                            0.0006983235944062471,
+                            0.00046871372614987195,
+                            0.0006973195122554898,
+                            0.0007315046386793256,
+                            0.0005920135881751776,
+                            0.0006415917305275798,
+                            0.0006416974356397986,
+                            0.000660997349768877,
+                            0.0006755717913620174,
+                            0.0007409897516481578,
+                            0.0005738807958550751,
+                            0.0005644228658638895,
+                            0.0006226671976037323,
+                            0.0005169115029275417,
+                            0.0007787267095409334,
+                            0.0005485793808475137,
+                            0.0006718726363033056,
+                            0.0006231243023648858,
+                            0.0006219388451427221,
+                            0.0006514330161735415,
+                            0.0006208544364199042,
+                            0.0007421888294629753,
+                            0.000751201412640512,
+                            0.000530686229467392,
+                            0.000637073302641511,
+                            0.0006908915820531547,
+                            0.0006077289581298828,
+                            0.0007463208166882396,
+                            0.0007028353284113109,
+                            0.0005152138764970005,
+                            0.0008116852841340005,
+                            0.0006334774079732597,
+                            0.0005230021779425442,
+                            0.00048497141688130796,
+                            0.0006976567092351615,
+                            0.0005448532174341381,
+                            0.0006540394970215857,
+                            0.0008178935968317091,
+                            0.0007322774035856128,
+                            0.0005780625506304204,
+                            0.0007965515251271427,
+                            0.0006191011052578688,
+                            0.0007472640718333423,
+                            0.0005441867397166789,
+                            0.0005791038274765015,
+                            0.0005265272338874638,
+                            0.0006100080208852887,
+                            0.000735417241230607,
+                            0.0005749164265580475,
+                            0.0006417296244762838,
+                            0.0006158063770271838,
+                            0.0006744630518369377,
+                            0.0005604362231679261,
+                            0.0007516483892686665,
+                            0.0006457090494222939,
+                            0.0007020284538157284,
+                            0.0006091106333769858,
+                            0.0006197014008648694,
+                            0.0005447991425171494,
+                            0.0007673096260987222,
+                            0.000696517585311085,
+                            0.0005551599897444248,
+                            0.0006719164084643126,
+                            0.0005527655594050884,
+                            0.0007175655919127166,
+                            0.0006422426667995751,
+                            0.0005824343534186482,
+                            0.0006053525721654296,
+                            0.000686083163600415,
+                            0.0005791898583993316,
+                            0.0006762866396456957,
+                            0.0006841323338449001,
+                            0.0005216868594288826,
+                            0.0005370815051719546,
+                            0.0005907252780161798,
+                            0.000547364994417876,
+                            0.000693018373567611,
+                            0.0006547651719301939,
+                            0.0005265969084575772,
+                            0.0006073724362067878,
+                            0.0005630754167214036,
+                            0.0007693101069889963,
+                            0.0006572194979526103,
+                            0.0005402046954259276,
+                            0.0005220441962592304,
+                            0.0005778452032245696,
+                            0.0005929520120844245,
+                            0.0006373710348270833,
+                            0.0006133383722044528,
+                            0.0006132589187473059,
+                            0.0006497990689240396,
+                            0.0005638811853714287,
+                            0.0007188550080172718,
+                            0.00048611886450089514,
+                            0.0007011168054305017,
+                            0.0006194326560944319,
+                            0.000691223714966327,
+                            0.00048495232476852834,
+                            0.0005529825575649738,
+                            0.000640335725620389,
+                            0.0007016219315119088,
+                            0.0008999171550385654,
+                            0.0005777355981990695,
+                            0.0006118105375207961,
+                            0.000618180725723505,
+                            0.0007082262309268117,
+                            0.0006179410847835243,
+                            0.0005946989404037595,
+                            0.00051576376426965,
+                            0.0005188914947211742,
+                            0.0005617542192339897,
+                            0.0006213588058017194,
+                            0.0006287246360443532,
+                            0.0006271051242947578,
+                            0.0005941138952039182,
+                            0.0006450085202232003,
+                            0.000716984795872122,
+                            0.0005565452156588435,
+                            0.0005658906302414834,
+                            0.0006780834519304335,
+                            0.000750103616155684,
+                            0.000647406151983887,
+                            0.0006173520814627409,
+                            0.0005955238011665642,
+                            0.0006089425296522677,
+                            0.000565613154321909,
+                            0.0005513374344445765,
+                            0.0005555696552619338,
+                            0.0004994423361495137,
+                            0.0006540477043017745,
+                            0.0005328420083969831,
+                            0.000589236558880657,
+                            0.0006822673021815717,
+                            0.0007716533727943897,
+                            0.0005836972850374877,
+                            0.0007366039208136499,
+                            0.0006712982431054115,
+                            0.0006632768199779093,
+                            0.0005516589153558016,
+                            0.0005770903080701828,
+                            0.0005559615092352033,
+                            0.0006587568204849958,
+                            0.0005702212220057845,
+                            0.00058025709586218,
+                            0.0006308215088211,
+                            0.0005325668025761843,
+                            0.0005859952652826905,
+                            0.0006864334573037922,
+                            0.0006282809772528708,
+                            0.0005715687875635922,
+                            0.0006624608067795634,
+                            0.0006364696309901774,
+                            0.0005057256785221398,
+                            0.0007261849823407829,
+                            0.0005599697469733655,
+                            0.000709354062564671,
+                            0.0007652048952877522,
+                            0.0005580090219154954,
+                            0.0006181342178024352,
+                            0.0005530293565243483,
+                            0.0006061553140170872,
+                            0.000537215790245682,
+                            0.0006744429119862616,
+                            0.0006123054772615433,
+                            0.0006305054412223399,
+                            0.0005663948832079768,
+                            0.0006792091298848391,
+                            0.000654345378279686,
+                            0.0005614960682578385,
+                            0.0005092329229228199,
+                            0.0008456074283458292,
+                            0.0006145521765574813,
+                            0.0005790682625956833,
+                            0.0005832352908328176,
+                            0.0005764519446529448,
+                            0.0006147925741970539,
+                            0.000561628898140043,
+                            0.00048511126078665257,
+                            0.0005423508700914681,
+                            0.000558424973860383,
+                            0.000527763448189944,
+                            0.0006569516845047474,
+                            0.0006080011371523142,
+                            0.0005810236325487494,
+                            0.0006501642637886107,
+                            0.0005858440999872983,
+                            0.0005831571761518717,
+                            0.0006133645656518638,
+                            0.0005574269453063607,
+                            0.0005994102102704346,
+                            0.0006472512031905353,
+                            0.0005523554282262921,
+                            0.0005119865527376533,
+                            0.0005522632272914052,
+                            0.0007534160977229476,
+                            0.0006984187057241797,
+                            0.0006037172279320657,
+                            0.000620763748884201,
+                            0.0005836021155118942,
+                            0.00046609650598838925,
+                            0.0008009580778889358,
+                            0.0005857673822902143,
+                            0.0006357811507768929,
+                            0.0007784189656376839,
+                            0.0006624026573263109,
+                            0.0006145025254227221,
+                            0.0008010654710233212,
+                            0.0006828739424236119,
+                            0.0004654347721952945,
+                            0.000956329342443496,
+                            0.0004990131710655987,
+                            0.000499424641020596,
+                            0.0006594903534278274,
+                            0.0005866636638529599,
+                            0.0005449826130643487,
+                            0.0006475101108662784,
+                            0.0005669985548593104,
+                            0.0006590820848941803,
+                            0.00087310082744807,
+                            0.0005570818320848048,
+                            0.0005942166317254305,
+                            0.0005821233498863876,
+                            0.0005301330820657313,
+                            0.0005489269969984889,
+                            0.0005959978443570435,
+                            0.0008702522027306259,
+                            0.0006571232806891203,
+                            0.0006477961433120072,
+                            0.0005880392272956669,
+                            0.0005005041020922363,
+                            0.0005295919836498797,
+                            0.0005655024433508515,
+                            0.0006865693721920252,
+                            0.0006138503667898476,
+                            0.0006308858864940703,
+                            0.0005402727983891964,
+                            0.0005357067566365004,
+                            0.0006138320313766599,
+                            0.000649390509352088,
+                            0.0005035185604356229,
+                            0.0006524370401166379,
+                            0.0005753038567490876,
+                            0.0005379676586017013,
+                            0.0006000518915243447,
+                            0.000634903903119266,
+                            0.0005498968530446291,
+                            0.0007571524474769831,
+                            0.0005620081792585552,
+                            0.0005136736435815692,
+                            0.000671219895593822,
+                            0.0005634346161969006,
+                            0.000583636574447155,
+                            0.0005337254260666668,
+                            0.0005712856072932482,
+                            0.0006052507669664919,
+                            0.0005124532035551965,
+                            0.0006117096054367721,
+                            0.0005462984554469585,
+                            0.0006764045683667064,
+                            0.0007240874692797661,
+                            0.0004944662796333432,
+                            0.0006689174915663898,
+                            0.0006343257264234126,
+                            0.0006983515922911465,
+                            0.0005913318018428981,
+                            0.0006610615528188646,
+                            0.0006278533255681396,
+                            0.0005796034820377827,
+                            0.000565974332857877,
+                            0.0006054554251022637,
+                            0.0007990287849679589,
+                            0.0006600199267268181,
+                            0.000727469043340534,
+                            0.0005032508634030819,
+                            0.0006389362970367074,
+                            0.0004929433926008642,
+                            0.0007060806383378804,
+                            0.0006012056255713105,
+                            0.0006519774324260652,
+                            0.0006757605588063598,
+                            0.0006704792031086981,
+                            0.0006162274512462318,
+                            0.0005845121340826154,
+                            0.0006501754396595061,
+                            0.0005406559212133288,
+                            0.0005392971797846258,
+                            0.000525354000274092,
+                            0.000676225230563432,
+                            0.0005362473893910646,
+                            0.000578970939386636,
+                            0.0006272306782193482,
+                            0.0006065734778530896,
+                            0.0006527447258122265,
+                            0.0005755292950198054,
+                            0.000565146969165653,
+                            0.0007529933354817331,
+                            0.0007873917929828167,
+                            0.000703417812474072,
+                            0.0006690789014101028,
+                            0.0005779993371106684,
+                            0.0006296274368651211,
+                            0.0005982343573123217,
+                            0.0007205926813185215,
+                            0.0006015249527990818,
+                            0.0005743862711824477,
+                            0.0005833909381181002,
+                            0.0006070958916097879,
+                            0.0005114689120091498,
+                            0.000680646626278758,
+                            0.0007837175508029759,
+                            0.0007534243632107973,
+                            0.0005716526648029685,
+                            0.0005350514547899365,
+                            0.0006887726485729218,
+                            0.0005616009002551436,
+                            0.0006806428427807987,
+                            0.0005532331997528672,
+                            0.0006970354588702321,
+                            0.0006984571809880435,
+                            0.0008790076244622469,
+                            0.0005149259814061224,
+                            0.0006484213517978787,
+                            0.0005613157409243286,
+                            0.0006457286071963608,
+                            0.0006438997224904597,
+                            0.0005595941329374909,
+                            0.0006930209347046912,
+                            0.0006198624614626169,
+                            0.0005585165345109999,
+                            0.0006681385566480458,
+                            0.0006013333331793547,
+                            0.0007202746346592903,
+                            0.0005698079476132989,
+                            0.0006851492798887193,
+                            0.000546466966625303,
+                            0.0005794817116111517,
+                            0.0006593995494768023,
+                            0.0007394214626401663,
+                            0.0005437797517515719,
+                            0.0006484111654572189,
+                            0.0006849571946077049,
+                            0.0006682599196210504,
+                            0.0005583775346167386,
+                            0.000726028869394213,
+                            0.0007769414805807173,
+                            0.000647359003778547,
+                            0.000775540538597852,
+                            0.0005351667641662061,
+                            0.0007281237049028277,
+                            0.0007158037042245269,
+                            0.0006064167828299105,
+                            0.0004952477756887674,
+                            0.0006304585840553045,
+                            0.0007616007351316512,
+                            0.0006322223343886435,
+                            0.0007915812893770635,
+                            0.0008975345408543944,
+                            0.0006162727950140834,
+                            0.0005726177478209138,
+                            0.0005298609612509608,
+                            0.0005793869495391846,
+                            0.0007024696678854525,
+                            0.0007163690752349794,
+                            0.0005938509711995721,
+                            0.0006180156487971544,
+                            0.0006418171687982976,
+                            0.0005624840268865228,
+                            0.0005709232063964009,
+                            0.0006351369083859026,
+                            0.0005673270206898451,
+                            0.0006212853477336466,
+                            0.0006189447012729943,
+                            0.0007161715184338391,
+                            0.0006022559246048331,
+                            0.0005728178657591343,
+                            0.0006124861538410187,
+                            0.0004975231713615358,
+                            0.0007002408965490758,
+                            0.0006072334945201874,
+                            0.0006143246428109705,
+                            0.0006940294988453388,
+                            0.0005307276733219624,
+                            0.0006341948756016791,
+                            0.0006067833746783435,
+                            0.0006083810585550964,
+                            0.0006295677158050239,
+                            0.0006761898985132575,
+                            0.0008461562683805823,
+                            0.0007371025858446956,
+                            0.0007478888146579266,
+                            0.0005325023666955531,
+                            0.000697785580996424,
+                            0.0007002093479968607,
+                            0.0005811757873743773,
+                            0.000579388695769012,
+                            0.0006268967990763485,
+                            0.0005820115911774337,
+                            0.000792847597040236,
+                            0.0006033963290974498,
+                            0.0005896277725696564,
+                            0.0007319090655073524,
+                            0.000573424156755209,
+                            0.0006870241486467421,
+                            0.0009746321593411267,
+                            0.0006572330021299422,
+                            0.0006459574215114117,
+                            0.0006239830981940031,
+                            0.0006152215646579862,
+                            0.0005988766206428409,
+                            0.0005706531228497624,
+                            0.0008580111316405237,
+                            0.0006402444560080767,
+                            0.0007859843317419291,
+                            0.0005551761714741588,
+                            0.0005853337934240699,
+                            0.0006696945638395846,
+                            0.0005643524928018451,
+                            0.0005755969323217869,
+                            0.0006683241226710379,
+                            0.0005270716501399875,
+                            0.00045238606981001794,
+                            0.0005409419536590576,
+                            0.0006158259348012507,
+                            0.0006635150639340281,
+                            0.0006276967469602823,
+                            0.0005811479059047997,
+                            0.0009667150443419814,
+                            0.0006321104010567069,
+                            0.0005579550634138286,
+                            0.0005102442810311913,
+                            0.000660003162920475,
+                            0.0012813638895750046,
+                            0.0007681092829443514,
+                            0.000596613681409508,
+                            0.0005572432419285178,
+                            0.0005741296918131411,
+                            0.0005962564027868211,
+                            0.0005673211417160928,
+                            0.0006096769939176738,
+                            0.0006412466173060238,
+                            0.0007180676911957562,
+                            0.000629797694273293,
+                            0.0006751304608769715,
+                            0.000625495973508805,
+                            0.0006277717766351998,
+                            0.0006129794637672603,
+                            0.0006916724960319698,
+                            0.000517007487360388,
+                            0.0005196101265028119,
+                            0.0005488035967573524,
+                            0.0006429736968129873,
+                            0.0005822963430546224,
+                            0.0006420820136554539,
+                            0.0006227968260645866,
+                            0.0005722242058254778,
+                            0.0007907702238298953,
+                            0.00073046307079494,
+                            0.0005801914958283305,
+                            0.0006537393783219159,
+                            0.0005503523861989379,
+                            0.0007023182115517557,
+                            0.0006456105620600283,
+                            0.0005858236690983176,
+                            0.0006988195236772299,
+                            0.000610769959166646,
+                            0.0006314042839221656,
+                            0.000734156696125865,
+                            0.0005378618952818215,
+                            0.00059428287204355,
+                            0.0007614323985762894,
+                            0.0005383134121075273,
+                            0.000538870575837791,
+                            0.0007125564734451473,
+                            0.0006149334367364645,
+                            0.000703215308021754,
+                            0.0006116608274169266,
+                            0.0009854943491518497,
+                            0.0005497634410858154,
+                            0.0007620479445904493,
+                            0.0006628856644965708,
+                            0.0005024455022066832,
+                            0.0005796694895252585,
+                            0.000678500859066844,
+                            0.0007821361068636179,
+                            0.0006412932998500764,
+                            0.0006099751335568726,
+                            0.0005111100617796183,
+                            0.0007626550504937768,
+                            0.0005461753462441266,
+                            0.0006180295604281127,
+                            0.0006828958285041153,
+                            0.000609255104791373,
+                            0.0006184650119394064,
+                            0.0005265175714157522,
+                            0.0006895210826769471,
+                            0.0006496576825156808,
+                            0.0005582065205089748,
+                            0.0006049092044122517,
+                            0.0004994319751858711,
+                            0.0005727637908421457,
+                            0.0006658229394815862,
+                            0.0005368278361856937,
+                            0.0005937280948273838,
+                            0.0005516510573215783,
+                            0.0004712441877927631,
+                            0.0005881073884665966,
+                            0.0006004226743243635,
+                            0.0004864538204856217,
+                            0.0005670781247317791,
+                            0.0006871614023111761,
+                            0.0007088605198077857,
+                            0.0005346558755263686,
+                            0.0007571025635115802,
+                            0.0006927598733454943,
+                            0.0007398736197501421,
+                            0.0005471411277540028,
+                            0.000539769243914634,
+                            0.0005950818303972483,
+                            0.0005579345743171871,
+                            0.0006694169715046883,
+                            0.0007180471438914537,
+                            0.0006046007620170712,
+                            0.0005843932740390301,
+                            0.0006484874174930155,
+                            0.0008058394305408001,
+                            0.0006330513278953731,
+                            0.0005841185338795185,
+                            0.0005242370534688234,
+                            0.0005105214077048004,
+                            0.0005868832231499255,
+                            0.0006034825928509235,
+                            0.0006212493753992021,
+                            0.000600344268605113,
+                            0.0006860419525764883,
+                            0.0006498898728750646,
+                            0.0005512865609489381,
+                            0.0006732014589942992,
+                            0.0006234191823750734,
+                            0.0005994318053126335,
+                            0.0006287341238930821,
+                            0.0005953186191618443,
+                            0.0005739968619309366,
+                            0.0006437430856749415,
+                            0.0004765397170558572,
+                            0.0005639806040562689,
+                            0.0007091052830219269,
+                            0.0005463366978801787,
+                            0.0007460624328814447,
+                            0.0006768713355995715,
+                            0.0005329717532731593,
+                            0.0006052919197827578,
+                            0.0007003927603363991,
+                            0.0005428371950984001,
+                            0.0007036210154183209,
+                            0.0006919559091329575,
+                            0.0006129555404186249,
+                            0.0007569534354843199,
+                            0.000632385490462184,
+                            0.0005478113889694214,
+                            0.000539823027793318,
+                            0.0006110711256042123,
+                            0.0006468324572779238,
+                            0.0005847951979376376,
+                            0.0005762892542406917,
+                            0.0005912137567065656,
+                            0.0006317851366475224,
+                            0.0006427439511753619,
+                            0.0006350322510115802,
+                            0.0006230776780284941,
+                            0.0006446182378567755,
+                            0.000664206687361002,
+                            0.0006735911010764539,
+                            0.0004783537006005645,
+                            0.0005844283150509,
+                            0.000609895505476743,
+                            0.000667972257360816,
+                            0.00048272000276483595,
+                            0.0005483256536535919,
+                            0.0005793423624709249,
+                            0.0006468681967817247,
+                            0.0005547003238461912,
+                            0.0005348632112145424,
+                            0.0005893102497793734,
+                            0.000505609146784991,
+                            0.0005228338413871825,
+                            0.0005880180397070944,
+                            0.0005798482452519238,
+                            0.0006059997249394655,
+                            0.000670426175929606,
+                            0.0006718541844747961,
+                            0.0005405771662481129,
+                            0.0005051325424574316,
+                            0.0005105044692754745,
+                            0.000564389571081847,
+                            0.0007792581454850733,
+                            0.0006504327757284045,
+                            0.000536124047357589,
+                            0.0004998894291929901,
+                            0.0006567646632902324,
+                            0.0006700907833874226,
+                            0.0006419180426746607,
+                            0.0005405812989920378,
+                            0.0006399370031431317,
+                            0.0005191110540181398,
+                            0.0005738987820222974,
+                            0.0007330309017561376,
+                            0.0005550325149670243,
+                            0.0004938157508149743,
+                            0.0006034908583387733,
+                            0.0006480110459960997,
+                            0.0005773508455604315,
+                            0.0006310814642347395,
+                            0.0005740729975514114,
+                            0.0006100888713262975,
+                            0.000659639947116375,
+                            0.0005380047368817031,
+                            0.0006269695004448295,
+                            0.0005450660828500986,
+                            0.0005986941978335381,
+                            0.0006160504417493939,
+                            0.0006234703469090164,
+                            0.0006876293919049203,
+                            0.0006114401621744037,
+                            0.0007923755329102278,
+                            0.0005299941985867918,
+                            0.0004994554328732193,
+                            0.0006973210256546736,
+                            0.00048114563105627894,
+                            0.0005366721889004111,
+                            0.0005630828673020005,
+                            0.0005290679982863367,
+                            0.0004866653180215508,
+                            0.0005671130493283272,
+                            0.00048423968837596476,
+                            0.0005503876600414515,
+                            0.0006186237442307174,
+                            0.0006760776159353554,
+                            0.0006295790080912411,
+                            0.0005281157209537923,
+                            0.0006532895495183766,
+                            0.0005164023605175316,
+                            0.0006304711569100618,
+                            0.000587292539421469,
+                            0.0005446899449452758,
+                            0.0005549151683226228,
+                            0.0005588004132732749,
+                            0.0005385075346566737,
+                            0.0006362833082675934,
+                            0.000579657091293484,
+                            0.000606888032052666,
+                            0.0006802074494771659,
+                            0.0005616911803372204,
+                            0.0005375315085984766,
+                            0.0006783806020393968,
+                            0.0005911887274123728,
+                            0.0005238123703747988,
+                            0.0006142233032733202,
+                            0.0004940059734508395,
+                            0.0006677728961221874,
+                            0.0005484121502377093,
+                            0.000564092886634171,
+                            0.0007282653241418302,
+                            0.0006626187241636217,
+                            0.0005173850222490728,
+                            0.000489523692522198,
+                            0.0006128391833044589,
+                            0.0006916679558344185,
+                            0.0006212700391188264,
+                            0.0005411060992628336,
+                            0.0005782237858511508,
+                            0.0004938143538311124,
+                            0.0006243021343834698,
+                            0.0006338309613056481,
+                            0.0006584118236787617,
+                            0.0006190112326294184,
+                            0.0006203587399795651,
+                            0.0005591948283836246,
+                            0.0006176934111863375,
+                            0.0006197753828018904,
+                            0.0006175488815642893,
+                            0.0007501401705667377,
+                            0.000597788137383759,
+                            0.0005391533486545086,
+                            0.0006836975808255374,
+                            0.000654720060992986,
+                            0.000542314606718719,
+                            0.0005596711998805404,
+                            0.0005585516337305307,
+                            0.0006345380679704249,
+                            0.0006647838745266199,
+                            0.0004525231197476387,
+                            0.000613694021012634,
+                            0.0007589001324959099,
+                            0.0005787800764665008,
+                            0.0006056270212866366,
+                            0.0006649108836427331,
+                            0.0007566764834336936,
+                            0.0005780252977274358,
+                            0.0005681401817128062,
+                            0.0005426360294222832,
+                            0.0006136945448815823,
+                            0.000690232147462666,
+                            0.0006018514977768064,
+                            0.0005741696804761887,
+                            0.0006083451444283128,
+                            0.0005199190927669406,
+                            0.0006289559532888234,
+                            0.0006487006903626025,
+                            0.0005495561053976417,
+                            0.0005374362808652222,
+                            0.0006862350855953991,
+                            0.0005569452187046409,
+                            0.0005647974903695285,
+                            0.0006388373440131545,
+                            0.000563110806979239,
+                            0.0006423074519261718,
+                            0.0005852689500898123,
+                            0.0007704788004048169,
+                            0.0005560819408856332,
+                            0.0005803858512081206,
+                            0.000706723309122026,
+                            0.0006520560127682984,
+                            0.0007073084125295281,
+                            0.0005359771312214434,
+                            0.0006200786447152495,
+                            0.0006917563150636852,
+                            0.0006355489022098482,
+                            0.0005058771348558366,
+                            0.0005694659776054323,
+                            0.0005770682473666966,
+                            0.0005953224026598036,
+                            0.0005652920226566494,
+                            0.0005632969550788403,
+                            0.0005230429815128446,
+                            0.0006665259134024382,
+                            0.000716866459697485,
+                            0.0005592487286776304,
+                            0.0005981222493574023,
+                            0.0008194803376682103,
+                            0.0006284257397055626,
+                            0.0005538540426641703,
+                            0.000652704038657248,
+                            0.0006052739918231964,
+                            0.000532406906131655,
+                            0.0006393485818989575,
+                            0.0005295926239341497,
+                            0.0005630861269310117,
+                            0.0006116846343502402,
+                            0.0005659459275193512,
+                            0.0005390014848671854,
+                            0.0006250354344956577,
+                            0.0005033797933720052,
+                            0.0007342022727243602,
+                            0.0005746633978560567,
+                            0.0004825418582186103,
+                            0.00048749649431556463,
+                            0.0008684121421538293,
+                            0.0006901639280840755,
+                            0.0005603868048638105,
+                            0.0005980637506581843,
+                            0.0005478705279529095,
+                            0.0005625206977128983,
+                            0.0005487434100359678,
+                            0.0005146954208612442,
+                            0.0005801002844236791,
+                            0.0005011003231629729,
+                            0.0005452838377095759,
+                            0.0005191824748180807,
+                            0.0005035727517679334,
+                            0.0005244117928668857,
+                            0.0005650946404784918,
+                            0.0005291514680720866,
+                            0.0006277637439779937,
+                            0.0005081159179098904,
+                            0.000524799048434943,
+                            0.0006233406020328403,
+                            0.0006420587305910885,
+                            0.000535974686499685,
+                            0.0005806529079563916,
+                            0.0005893950583413243,
+                            0.000541395042091608,
+                            0.0005856260540895164,
+                            0.0005389683647081256,
+                            0.0005201957537792623,
+                            0.0005832198075950146,
+                            0.0005427887081168592,
+                            0.0006160189514048398,
+                            0.0006038225837983191,
+                            0.0007280526333488524,
+                            0.0005491517367772758,
+                            0.0005912131746299565,
+                            0.0005227699293754995,
+                            0.0005278567550703883,
+                            0.0006401250138878822,
+                            0.0005240592872723937,
+                            0.0005249844980426133,
+                            0.0006974272546358407,
+                            0.0006661636871285737,
+                            0.0005042572738602757,
+                            0.0006021995795890689,
+                            0.0006813997752033174,
+                            0.0006020513246767223,
+                            0.0004925601533614099,
+                            0.0006195164169184864,
+                            0.0005258519086055458,
+                            0.000650861591566354,
+                            0.0005509425536729395,
+                            0.0006616163300350308,
+                            0.0004805399221368134,
+                            0.0005380006623454392,
+                            0.0005277019809000194,
+                            0.0006013275706209242,
+                            0.0005772504955530167,
+                            0.0006147677195258439,
+                            0.0006285684648901224,
+                            0.0006173477740958333,
+                            0.0006472869426943362,
+                            0.0005342094809748232,
+                            0.0006885108887217939,
+                            0.000566813803743571,
+                            0.0006678761565126479,
+                            0.0005599106079898775,
+                            0.0005195103003643453,
+                            0.0005005974671803415,
+                            0.0005407891585491598,
+                            0.000536752340849489,
+                            0.0007269185734912753,
+                            0.0005604306352324784,
+                            0.0006288812728598714,
+                            0.0005353695014491677,
+                            0.000515229010488838,
+                            0.0005537659162655473,
+                            0.0006129881949163973,
+                            0.0006255139014683664,
+                            0.0007105133263394237,
+                            0.0005741164204664528,
+                            0.0006328226299956441,
+                            0.0007649907493032515,
+                            0.0005711002741008997,
+                            0.0006857445696368814,
+                            0.0005448641604743898,
+                            0.0006013837410137057,
+                            0.0005174515536054969,
+                            0.0005537852412089705,
+                            0.00048288164543919265,
+                            0.0005192377138882875,
+                            0.0005416377680376172,
+                            0.0005086440360173583,
+                            0.0005024708225391805,
+                            0.0005166074843145907,
+                            0.0006004258757457137,
+                            0.0006998037570156157,
+                            0.0005765065434388816,
+                            0.0005210511153563857,
+                            0.0006150632398203015,
+                            0.0007407480152323842,
+                            0.0005419099470600486,
+                            0.0004662910650949925,
+                            0.0005826408741995692,
+                            0.00057232502149418,
+                            0.0005847668508067727,
+                            0.0005452153272926807,
+                            0.0006492850370705128,
+                            0.0006536614964716136,
+                            0.0005697883316315711,
+                            0.0006384173757396638,
+                            0.0006712867761962116,
+                            0.0005929396138526499,
+                            0.0006927134236320853,
+                            0.000626595807261765,
+                            0.0006018016138114035,
+                            0.0007718929555267096,
+                            0.0008332376019097865,
+                            0.0005871686153113842,
+                            0.0005300284828990698,
+                            0.0005784162785857916,
+                            0.0005907051963731647,
+                            0.0005684640491381288,
+                            0.000730277388356626,
+                            0.0007892026333138347,
+                            0.0006754141650162637,
+                            0.0006438291165977716,
+                            0.000578956794925034,
+                            0.0005411003367044032,
+                            0.0006149770342744887,
+                            0.0008610684890300035,
+                            0.0006397675606422126,
+                            0.0005193239194341004,
+                            0.0005688206292688847,
+                            0.0009441008442081511,
+                            0.0007115391199477017,
+                            0.0006166642415337265,
+                            0.0006096563884057105,
+                            0.0007638633251190186,
+                            0.0007637597736902535,
+                            0.00060824811225757,
+                            0.0005202918546274304,
+                            0.0005046332371421158,
+                            0.0005363691016100347,
+                            0.0006943433545529842,
+                            0.0006263555842451751,
+                            0.0005455520586110651,
+                            0.0006209269049577415,
+                            0.0006643543019890785,
+                            0.0007015978917479515,
+                            0.0006517345318570733,
+                            0.00047037057811394334,
+                            0.0006635038880631328,
+                            0.0007976908818818629,
+                            0.0005916722002439201,
+                            0.0005230600945651531,
+                            0.0007215038640424609,
+                            0.000583737506531179,
+                            0.0005786214605905116,
+                            0.0006167440442368388,
+                            0.0005869033630006015,
+                            0.0005640173330903053,
+                            0.0006173047004267573,
+                            0.0005813832976855338,
+                            0.0005670780665241182,
+                            0.0007478810148313642,
+                            0.0005096157547086477,
+                            0.0005948637262918055,
+                            0.000576907186768949,
+                            0.0006685506668873131,
+                            0.0006616990431211889,
+                            0.0006503493059426546,
+                            0.0007837564335204661,
+                            0.0006929895607754588,
+                            0.0006544535863213241,
+                            0.000544717360753566,
+                            0.0005497703677974641,
+                            0.0005801204824820161,
+                            0.0006703153485432267,
+                            0.0006426173495128751,
+                            0.0006456658593378961,
+                            0.000685848412103951,
+                            0.0007519147475250065,
+                            0.0005650157108902931,
+                            0.0007062041549943388,
+                            0.0006261655362322927,
+                            0.0006591774872504175,
+                            0.0005630352534353733,
+                            0.0006340739200823009,
+                            0.0005445388960652053,
+                            0.0005395594635047019,
+                            0.0006859991117380559,
+                            0.0005954905063845217,
+                            0.0006269239238463342,
+                            0.0005225147469900548,
+                            0.0008298921165987849,
+                            0.0005466222646646202,
+                            0.0006395450909622014,
+                            0.0006335536018013954,
+                            0.0005161194712854922,
+                            0.0005700925830751657,
+                            0.000710010586772114,
+                            0.000631471280939877,
+                            0.000611843541264534,
+                            0.0005978088593110442,
+                            0.0005851003807038069,
+                            0.0005108159966766834,
+                            0.0006228754064068198,
+                            0.0006411570939235389,
+                            0.0005518939578905702,
+                            0.0006481340387836099,
+                            0.0006578703760169446,
+                            0.0006228529382497072,
+                            0.0005971459904685616,
+                            0.0006229215650819242,
+                            0.0006784416036680341,
+                            0.0006281172391027212,
+                            0.001099838875234127,
+                            0.0005755611346103251,
+                            0.0006694091716781259,
+                            0.00048520517884753644,
+                            0.0005504846340045333,
+                            0.0006848007324151695,
+                            0.00053523009410128,
+                            0.0006420706049539149,
+                            0.0006867091869935393,
+                            0.0005764044472016394,
+                            0.0005973528022877872,
+                            0.0005052798660472035,
+                            0.0006184296798892319,
+                            0.0005752576398663223,
+                            0.0005299639888107777,
+                            0.0005649197264574468,
+                            0.0005724991788156331,
+                            0.0005838503129780293,
+                            0.0005588784115388989,
+                            0.0006432711379602551,
+                            0.0006166587118059397,
+                            0.0006340767722576857,
+                            0.0005954985390417278,
+                            0.0005437242798507214,
+                            0.0007651760824956,
+                            0.000506310781929642,
+                            0.0005767260445281863,
+                            0.0005683755152858794,
+                            0.0005470836185850203,
+                            0.0008190066437236965,
+                            0.000628278183285147,
+                            0.0004911949508823454,
+                            0.0005802064551971853,
+                            0.000523519644048065,
+                            0.000652383838314563,
+                            0.0006306966533884406,
+                            0.0006900721928104758,
+                            0.0005544325686059892,
+                            0.0007119622314348817,
+                            0.0005916535155847669,
+                            0.0006447110208682716,
+                            0.00046995392767712474,
+                            0.0005982542061246932,
+                            0.0005618928116746247,
+                            0.0007439152104780078,
+                            0.0006391779752448201,
+                            0.0005001349491067231,
+                            0.0007391080725938082,
+                            0.0006601240602321923,
+                            0.0006081628380343318,
+                            0.0006217915215529501,
+                            0.00066353939473629,
+                            0.0005301829660311341,
+                            0.0006390591734088957,
+                            0.0006352330092340708,
+                            0.0005146210314705968,
+                            0.0006923085893504322,
+                            0.000552934652660042,
+                            0.0006105556385591626,
+                            0.0006013073725625873,
+                            0.0005937075475230813,
+                            0.0005950548802502453,
+                            0.0005363482632674277,
+                            0.0005781009094789624,
+                            0.000577228085603565,
+                            0.0004984547267667949,
+                            0.0006153383874334395,
+                            0.0005162275629118085,
+                            0.000741328694857657,
+                            0.0006000621942803264,
+                            0.0006172604626044631,
+                            0.0005357829504646361,
+                            0.0007398020243272185,
+                            0.0005654633278027177,
+                            0.0007017613388597965,
+                            0.0006916642305441201,
+                            0.0005245518404990435,
+                            0.0005617034039460123,
+                            0.0005759352934546769,
+                            0.0005865885177627206,
+                            0.0006078454898670316,
+                            0.0006797961541451514,
+                            0.0005828525172546506,
+                            0.0005525031010620296,
+                            0.0007279454148374498,
+                            0.0005668005906045437,
+                            0.0009171418496407568,
+                            0.0005274926079437137,
+                            0.0005146028124727309,
+                            0.0005628577200695872,
+                            0.0005751948338001966,
+                            0.0005977825494483113,
+                            0.0006444922182708979,
+                            0.000570550502743572,
+                            0.0004815561987925321,
+                            0.0006859313580207527,
+                            0.0006659763748757541,
+                            0.0006163973594084382,
+                            0.000450899766292423,
+                            0.0005215129349380732,
+                            0.0006529953680001199,
+                            0.00045831126044504344,
+                            0.0005559763521887362,
+                            0.0005971429636701941,
+                            0.0006460176664404571,
+                            0.00054381467634812,
+                            0.0005977399996481836,
+                            0.0006224172539077699,
+                            0.0006641443469561636,
+                            0.0005998977576382458,
+                            0.0005666935467161238,
+                            0.0007084393291734159,
+                            0.0005355994799174368,
+                            0.000637863646261394,
+                            0.0005109167541377246,
+                            0.0005877699586562812,
+                            0.0005353230517357588,
+                            0.0006418701959773898,
+                            0.0005961732240393758,
+                            0.0005946538876742125,
+                            0.000606209272518754,
+                            0.0005866619758307934,
+                            0.0006356051890179515,
+                            0.0005348006961867213,
+                            0.0005876737996004522,
+                            0.0008046839502640069,
+                            0.0005527905304916203,
+                            0.0005585700855590403,
+                            0.0005301719647832215,
+                            0.0005347900441847742,
+                            0.0005756557220593095,
+                            0.000718510418664664,
+                            0.0005396493943408132,
+                            0.0005306127131916583,
+                            0.0006549520185217261,
+                            0.0007168345618993044,
+                            0.0006923440960235894,
+                            0.000582417706027627,
+                            0.0005732611753046513,
+                            0.0006093434640206397,
+                            0.0006862803129479289,
+                            0.0006373987416736782,
+                            0.0006677127676084638,
+                            0.00065091613214463,
+                            0.0005563515587709844,
+                            0.0006468739011324942,
+                            0.0007407264201901853,
+                            0.0005780785577371716,
+                            0.0007615804788656533,
+                            0.0005652940017171204,
+                            0.000591909047216177,
+                            0.0005436853389255702,
+                            0.0005493140779435635,
+                            0.000504009542055428,
+                            0.0006321530090644956,
+                            0.0005307645769789815,
+                            0.0005287298699840903,
+                            0.0005241454346105456,
+                            0.0006158355972729623,
+                            0.000518104643560946,
+                            0.0006215223693288863,
+                            0.0005442285328172147,
+                            0.0005230024689808488,
+                            0.0006193675799295306,
+                            0.0005504423170350492,
+                            0.0007704495801590383,
+                            0.0005107289180159569,
+                            0.000554732047021389,
+                            0.0006056675338186324,
+                            0.0005273928982205689,
+                            0.0005208780057728291,
+                            0.0005525457672774792,
+                            0.0006181913777254522,
+                            0.000596919097006321,
+                            0.000534995342604816,
+                            0.000592216441873461,
+                            0.000572226766962558,
+                            0.0004946797271259129,
+                            0.0006362671847455204,
+                            0.0006031335215084255,
+                            0.0005917925736866891,
+                            0.0005413147737272084,
+                            0.0005488712340593338,
+                            0.0006245470140129328,
+                            0.0005920441471971571,
+                            0.0005660029128193855,
+                            0.0005957975517958403,
+                            0.0005330429412424564,
+                            0.0006092213443480432,
+                            0.0005491559277288616,
+                            0.0005388893187046051,
+                            0.0005541207501664758,
+                            0.0004630338808055967,
+                            0.0005391898448579013,
+                            0.0006658336496911943,
+                            0.0004958074423484504,
+                            0.0005769049166701734,
+                            0.0006550051039084792,
+                            0.00048599287401884794,
+                            0.0006273768376559019,
+                            0.0006938747828826308,
+                            0.0005171725642867386,
+                            0.0006197497132234275,
+                            0.0005822453531436622,
+                            0.00055525335483253,
+                            0.0005515387165360153,
+                            0.0005362975061871111,
+                            0.0006074497359804809,
+                            0.0005859981756657362,
+                            0.0006346352747641504,
+                            0.0007109539001248777,
+                            0.0005710461409762502,
+                            0.0006088243098929524,
+                            0.0008102545398287475,
+                            0.0005127892945893109,
+                            0.0005454914644360542,
+                            0.0006530637620016932,
+                            0.000582138542085886,
+                            0.0005921326810494065,
+                            0.0005102638970129192,
+                            0.0005777156329713762,
+                            0.0006051887758076191,
+                            0.0007028889958746731,
+                            0.0005850749439559877,
+                            0.000519916822668165,
+                            0.0006608629482798278,
+                            0.0006073107360862195,
+                            0.0004769175429828465,
+                            0.0005665384232997894,
+                            0.0005679025780409575,
+                            0.0005312674911692739,
+                            0.0005594997783191502,
+                            0.0006130679976195097,
+                            0.0005680213798768818,
+                            0.0006281732930801809,
+                            0.0006286984425969422,
+                            0.0006914083496667445,
+                            0.0005463249399326742,
+                            0.0005518217221833766,
+                            0.0005817476194351912,
+                            0.0005814568721689284,
+                            0.0006175418384373188,
+                            0.0006027971394360065,
+                            0.000581146334297955,
+                            0.0005663863266818225,
+                            0.0006510780658572912,
+                            0.0006546410731971264,
+                            0.0005803022067993879,
+                            0.0005073921056464314,
+                            0.000657097261864692,
+                            0.0005505584995262325,
+                            0.00054845807608217,
+                            0.0007062553777359426,
+                            0.0005285137449391186,
+                            0.0005876616924069822,
+                            0.0005797530757263303,
+                            0.0005878172814846039,
+                            0.0005444150301627815,
+                            0.0005716383457183838,
+                            0.0005296032759360969,
+                            0.0006712962058372796,
+                            0.0006286056013777852,
+                            0.0005404289695434272,
+                            0.0006902028108015656,
+                            0.0004779064911417663,
+                            0.0005344583187252283,
+                            0.0007108067511580884,
+                            0.0006280109519138932,
+                            0.0005308425170369446,
+                            0.0005878686206415296,
+                            0.0005569735076278448,
+                            0.0005613719113171101,
+                            0.0006812725914642215,
+                            0.00049323879648,
+                            0.0005660646711476147,
+                            0.0005250992835499346,
+                            0.0005949065671302378,
+                            0.0005190279334783554,
+                            0.0006510070525109768,
+                            0.0005942924180999398,
+                            0.0005480297841131687,
+                            0.0005516285309568048,
+                            0.0006450066575780511,
+                            0.00047844674554653466,
+                            0.0007502465741708875,
+                            0.0006031125667504966,
+                            0.0004985373234376311,
+                            0.0007277206750586629,
+                            0.0005655826535075903,
+                            0.0005643184413202107,
+                            0.000555599108338356,
+                            0.0006617972976528108,
+                            0.0005706044030375779,
+                            0.0005481959087774158,
+                            0.0006949889357201755,
+                            0.0006180485361255705,
+                            0.0005903863348066807,
+                            0.0006102398619987071,
+                            0.0006165818776935339,
+                            0.0005987955955788493,
+                            0.00048434865311719477,
+                            0.0006978267338126898,
+                            0.0005255513824522495,
+                            0.0005739561165682971,
+                            0.0006847700569778681,
+                            0.0005019568488933146,
+                            0.0005060550756752491,
+                            0.0005335842724889517,
+                            0.0006203080411069095,
+                            0.000678454409353435,
+                            0.0005297836614772677,
+                            0.0007263966836035252,
+                            0.0006908699288032949,
+                            0.0005515335942618549,
+                            0.0005754531593993306,
+                            0.0006681859958916903,
+                            0.000633056799415499,
+                            0.0005588107742369175,
+                            0.0005755782476626337,
+                            0.0006367088644765317,
+                            0.0005211820243857801,
+                            0.0005871782195754349,
+                            0.0006041160668246448,
+                            0.000575087673496455,
+                            0.0006272654281929135,
+                            0.0004948179703205824,
+                            0.0005807182169519365,
+                            0.000624262320343405,
+                            0.0005778006743639708,
+                            0.0005425389972515404,
+                            0.0005806830013170838,
+                            0.0006541954935528338,
+                            0.0005452869809232652,
+                            0.00044982886174693704,
+                            0.0005118473782204092,
+                            0.0006291873287409544,
+                            0.0007344273617491126,
+                            0.0006210631690919399,
+                            0.0005644321208819747,
+                            0.0006102888728491962,
+                            0.0005070316256023943,
+                            0.0005997795378789306,
+                            0.0006004461320117116,
+                            0.0006655343458987772,
+                            0.0008060563122853637,
+                            0.000505606469232589,
+                            0.000587796384934336,
+                            0.0005870065069757402,
+                            0.0006205924437381327,
+                            0.0006066361675038934,
+                            0.0005401114467531443,
+                            0.0006439812132157385,
+                            0.0006072710384614766,
+                            0.0005361170042306185,
+                            0.0007744805770926178,
+                            0.0005040623946115375,
+                            0.0007134274928830564,
+                            0.0005728222313337028,
+                            0.000560820335522294,
+                            0.0006088507943786681,
+                            0.0005883440608158708,
+                            0.0005441870307549834,
+                            0.000705976621247828,
+                            0.0005022311815991998,
+                            0.0004949121503159404,
+                            0.0005564724560827017,
+                            0.0006623218068853021,
+                            0.0005942088318988681,
+                            0.0005762811633758247,
+                            0.0005866941646672785,
+                            0.000630655325949192,
+                            0.0005319671472534537,
+                            0.0005891637410968542,
+                            0.0006516706780530512,
+                            0.0006314247148111463,
+                            0.0004872587451245636,
+                            0.0007320123841054738,
+                            0.0005363849923014641,
+                            0.0005220024031586945,
+                            0.0007289046188816428,
+                            0.0006743571721017361,
+                            0.0006845768075436354,
+                            0.0005280919140204787,
+                            0.0006371085764840245,
+                            0.0005052571650594473,
+                            0.0006613389123231173,
+                            0.000638331868685782,
+                            0.0004724158497992903,
+                            0.0007604334386996925,
+                            0.0006896682898513973,
+                            0.0007799426675774157,
+                            0.0004813145787920803,
+                            0.0007119151996448636,
+                            0.0006538691814057529,
+                            0.0005158645799383521,
+                            0.0005753536825068295,
+                            0.000708605395630002,
+                            0.0005684581119567156,
+                            0.0005439492524601519,
+                            0.0005388507852330804,
+                            0.0004749881336465478,
+                            0.0005641856114380062,
+                            0.0005700511974282563,
+                            0.0005696106236428022,
+                            0.0006148609099909663,
+                            0.000611830735579133,
+                            0.0005616251146420836,
+                            0.0006211493746377528,
+                            0.0005652813706547022,
+                            0.0005650303210131824,
+                            0.0007443774375133216,
+                            0.0006247383425943553,
+                            0.0007723143789917231,
+                            0.0005350240389816463,
+                            0.0006902684108354151,
+                            0.0005820862133987248,
+                            0.0006029948126524687,
+                            0.0005849951412528753,
+                            0.0005480275140143931,
+                            0.0006089668022468686,
+                            0.000603625550866127,
+                            0.0007018574979156256,
+                            0.0006240875809453428,
+                            0.0006536951987072825,
+                            0.0005754465237259865,
+                            0.0005333532462827861,
+                            0.000645600026473403,
+                            0.0005927616730332375,
+                            0.0006083414773456752,
+                            0.0006731172325089574,
+                            0.0007518708589486778,
+                            0.0006286763236857951,
+                            0.0008374524768441916,
+                            0.0008048631716519594,
+                            0.0006537038716487586,
+                            0.0005900379037484527,
+                            0.000619006808847189,
+                            0.0005694386782124639,
+                            0.000609695038292557,
+                            0.0006549148820340633,
+                            0.0007105027325451374,
+                            0.0006499342853203416,
+                            0.0005372334271669388,
+                            0.0005086050368845463,
+                            0.0007027055253274739,
+                            0.0006266066338866949,
+                            0.0007743543246760964,
+                            0.0005376478657126427,
+                            0.00069231673842296,
+                            0.0006787003367207944,
+                            0.000659769051708281,
+                            0.0005347981350496411,
+                            0.0006182905635796487,
+                            0.0006478046998381615,
+                            0.0005631530657410622,
+                            0.0007697064429521561,
+                            0.000555479375179857,
+                            0.0006114614079706371,
+                            0.0005940879927948117,
+                            0.0006795736844651401,
+                            0.0007211127085611224,
+                            0.000717740913387388,
+                            0.0007080335053615272,
+                            0.00069275131681934,
+                            0.0006806571036577225,
+                            0.0006461091688834131,
+                            0.000719554431270808,
+                            0.0005806381814181805,
+                            0.0005846591084264219,
+                            0.0007396641303785145,
+                            0.0005955292726866901,
+                            0.0005862257094122469,
+                            0.0005957544199191034,
+                            0.0006946478970348835,
+                            0.000596916361246258,
+                            0.0005046377773396671,
+                            0.000619355239905417,
+                            0.0006038007559254766,
+                            0.0004993774928152561,
+                            0.0006138847675174475,
+                            0.0007835259893909097,
+                            0.000495866930577904,
+                            0.000583004264626652,
+                            0.0005359018687158823,
+                            0.0008075552759692073,
+                            0.000610302435234189,
+                            0.0007320807781070471,
+                            0.0007983514806255698,
+                            0.0005856009083800018,
+                            0.0006967437220737338,
+                            0.0005711278063245118,
+                            0.0005818376084789634,
+                            0.0006729951128363609,
+                            0.0005967649049125612,
+                            0.0005698890890926123,
+                            0.0005522760911844671,
+                            0.00075729243690148,
+                            0.0007317822310142219,
+                            0.0006927622598595917,
+                            0.0006137096206657588,
+                            0.0005603019380941987,
+                            0.000555232516489923,
+                            0.0005934082437306643,
+                            0.0006757503724656999,
+                            0.0006704003899358213,
+                            0.0006323522538878024,
+                            0.0005841900710947812,
+                            0.0005977099644951522,
+                            0.0006046521593816578,
+                            0.000698001473210752,
+                            0.0006143201608210802,
+                            0.0006159892654977739,
+                            0.0006896215490996838,
+                            0.0007904815138317645,
+                            0.0006359957624226809,
+                            0.0006198424962349236,
+                            0.0006753897177986801,
+                            0.0005429128068499267,
+                            0.0005555455572903156,
+                            0.0005917687085457146,
+                            0.0007076264009810984,
+                            0.0005925385630689561,
+                            0.0006790186162106693,
+                            0.0006764568970538676,
+                            0.0006243156385608017,
+                            0.0006212380831129849,
+                            0.0005996964173391461,
+                            0.000764234340749681,
+                            0.0006328016170300543,
+                            0.0006352381315082312,
+                            0.0005858815857209265,
+                            0.0006402703002095222,
+                            0.0006283872644416988,
+                            0.0006623067311011255,
+                            0.0006797067471779883,
+                            0.0004856144660152495,
+                            0.0004878361360169947,
+                            0.0005151854129508138,
+                            0.0005901770200580359,
+                            0.0006054340628907084,
+                            0.0006424118764698505,
+                            0.0007017070311121643,
+                            0.0011146244360134006,
+                            0.0007875556475482881,
+                            0.0006062745815142989,
+                            0.0006823809817433357,
+                            0.0006067376816645265,
+                            0.000949983368627727,
+                            0.0006755994400009513,
+                            0.0005803856183774769,
+                            0.000752450549043715,
+                            0.0005511601339094341,
+                            0.0008104440057650208,
+                            0.0009188445401377976,
+                            0.0007282024016603827,
+                            0.0006351051852107048,
+                            0.0005732090794481337,
+                            0.0006969376699998975,
+                            0.0005187899223528802,
+                            0.0006517692236229777,
+                            0.0005686377407982945,
+                            0.0005690856487490237,
+                            0.0009094642591662705,
+                            0.0006372838979586959,
+                            0.0006723029655404389,
+                            0.0006834849482402205,
+                            0.000540355802513659,
+                            0.0007605711580254138,
+                            0.000550587777979672,
+                            0.0009416642133146524,
+                            0.0008408516296185553,
+                            0.0007322530727833509,
+                            0.0006256545311771333,
+                            0.000600486877374351,
+                            0.0006831313367001712,
+                            0.0010627623414620757,
+                            0.0007711398066021502,
+                            0.0005532786017283797,
+                            0.000748532242141664,
+                            0.0005581300356425345,
+                            0.000737773603759706,
+                            0.000780490692704916,
+                            0.0007166191353462636,
+                            0.000590843497775495,
+                            0.0008924061548896134,
+                            0.0008641401072964072,
+                            0.0005245372885838151,
+                            0.0005691721453331411,
+                            0.0006288859294727445,
+                            0.0006967830122448504,
+                            0.0010818765731528401,
+                            0.0005679359892383218,
+                            0.0005866067367605865,
+                            0.000675542454700917,
+                            0.000739379960577935,
+                            0.0005883196135982871,
+                            0.0007156124338507652,
+                            0.0007750732474960387,
+                            0.0007847679080441594,
+                            0.000517509295605123,
+                            0.0004882833454757929,
+                            0.0006918567814864218,
+                            0.0005758518818765879,
+                            0.0005172103992663324,
+                            0.0005173451500013471,
+                            0.0005345676327124238,
+                            0.000657568802125752,
+                            0.000703081488609314,
+                            0.0004527297569438815,
+                            0.0006799628608860075,
+                            0.0006102971965447068,
+                            0.0007503426168113947,
+                            0.001082511735148728,
+                            0.0006245115073397756,
+                            0.000676812487654388,
+                            0.0004970517475157976,
+                            0.0008108039619401097,
+                            0.0007503964006900787,
+                            0.0006230426370166242,
+                            0.0007565116393379867,
+                            0.0005932831554673612,
+                            0.0007220166735351086,
+                            0.0008160601719282568,
+                            0.0007208795286715031,
+                            0.0011012260802090168,
+                            0.0006071078823879361,
+                            0.0010137493954971433,
+                            0.0005715300794690847,
+                            0.0008617094135843217,
+                            0.0005604274338111281,
+                            0.0005659939488396049,
+                            0.000544100534170866,
+                            0.0006819719565100968,
+                            0.0007953757303766906,
+                            0.0005216908757574856,
+                            0.0006464666221290827,
+                            0.0005603045574389398,
+                            0.0006240906659513712,
+                            0.0011202221503481269,
+                            0.0009208372794091702,
+                            0.0010114365722984076,
+                            0.0008042091503739357,
+                            0.0005615549744106829,
+                            0.0007474313024431467,
+                            0.000749831204302609,
+                            0.0009522834443487227,
+                            0.0004734962712973356,
+                            0.0006061921012587845,
+                            0.0005862321122549474,
+                            0.0005491957999765873,
+                            0.0005927780875936151,
+                            0.0005064219585619867,
+                            0.0007254160009324551,
+                            0.0006038338178768754,
+                            0.000747981364838779,
+                            0.0005956406821496785,
+                            0.0007194034405983984,
+                            0.0006477784481830895,
+                            0.0007325323531404138,
+                            0.0006741913966834545,
+                            0.0006305758724920452,
+                            0.0005133830709382892,
+                            0.0006295505445450544,
+                            0.000660163233987987,
+                            0.000842875218950212,
+                            0.0006030192016623914,
+                            0.000627906818408519,
+                            0.0006077363505028188,
+                            0.0005842578248120844,
+                            0.0005950796185061336,
+                            0.0007006271625868976,
+                            0.0006216475157998502,
+                            0.0005279210745356977,
+                            0.000692018773406744,
+                            0.0006759712123312056,
+                            0.0006660912767983973,
+                            0.0005111848586238921,
+                            0.0006064851768314838,
+                            0.0006327315350063145,
+                            0.0005562584265135229,
+                            0.0005673964624293149,
+                            0.0004913313896395266,
+                            0.0007435819134116173,
+                            0.0005547028849832714,
+                            0.0006577596650458872,
+                            0.0005954111693426967,
+                            0.001015493064187467,
+                            0.0006711938185617328,
+                            0.0006060166633687913,
+                            0.0006552322302013636,
+                            0.000597936799749732,
+                            0.0006456697592511773,
+                            0.0005519789410755038,
+                            0.0005422086687758565,
+                            0.0008661276660859585,
+                            0.0006254619220271707,
+                            0.00046922630281187594,
+                            0.0006685761618427932,
+                            0.0005999645218253136,
+                            0.0006160637130960822,
+                            0.0005555619718506932,
+                            0.0007494557648897171,
+                            0.0009316211217083037,
+                            0.0006170197739265859,
+                            0.0005938911926932633,
+                            0.000608112895861268,
+                            0.0011318319011479616,
+                            0.0008884606650099158,
+                            0.0005503535503521562,
+                            0.0007497625192627311,
+                            0.0006850859499536455,
+                            0.0009501096210442483,
+                            0.0008942129206843674,
+                            0.0006500378949567676,
+                            0.0006479814765043557,
+                            0.0008309986442327499,
+                            0.0006130956462584436,
+                            0.0005909951287321746,
+                            0.0007346457568928599,
+                            0.0007534631877206266,
+                            0.0006865650648251176,
+                            0.0006163732032291591,
+                            0.0006438225391320884,
+                            0.00046696406207047403,
+                            0.0006257928325794637,
+                            0.0005797892226837575,
+                            0.0005151794175617397,
+                            0.0005669222446158528,
+                            0.0005441278917714953,
+                            0.000614700373262167,
+                            0.0005925910663791001,
+                            0.0005299811600707471,
+                            0.0006672625313512981,
+                            0.0005794765311293304,
+                            0.0007913392619229853,
+                            0.000817654246930033,
+                            0.0006273379549384117,
+                            0.0005443732952699065,
+                            0.0007954229367896914,
+                            0.0008412306779064238,
+                            0.0006804233998991549,
+                            0.0007377123110927641,
+                            0.0008825591648928821,
+                            0.0007031832938082516,
+                            0.0006105743814259768,
+                            0.0006209479179233313,
+                            0.0006276832427829504,
+                            0.0008054531062953174,
+                            0.0007461165077984333,
+                            0.0006940176826901734,
+                            0.0006669422728009522,
+                            0.0008469544118270278,
+                            0.0005685987416654825,
+                            0.0007223651628009975,
+                            0.0005798063357360661,
+                            0.0005738917971029878,
+                            0.0009908952051773667,
+                            0.0007784265908412635,
+                            0.0005999269196763635,
+                            0.0005825645057484508,
+                            0.0005200340528972447,
+                            0.0006077064317651093,
+                            0.0007215731311589479,
+                            0.0006087098154239357,
+                            0.0006697086500935256,
+                            0.0004944889224134386,
+                            0.0005306257517077029,
+                            0.0006592787685804069,
+                            0.0007432769052684307,
+                            0.0008703815401531756,
+                            0.0005996105610392988,
+                            0.000596271944232285,
+                            0.000513895764015615,
+                            0.000562397064641118,
+                            0.0006824549636803567,
+                            0.0005558427073992789,
+                            0.0005693834973499179,
+                            0.0008141276775859296,
+                            0.0009426638716831803,
+                            0.0005367643898352981,
+                            0.0007755928090773523,
+                            0.0006294975173659623,
+                            0.0007308454951271415,
+                            0.0005602180026471615,
+                            0.0005929372855462134,
+                            0.0006388186593540013,
+                            0.0007423429051414132,
+                            0.0005623813485726714,
+                            0.0005853999173268676,
+                            0.0009053200483322144,
+                            0.0005596260307356715,
+                            0.0006121970363892615,
+                            0.000897422491107136,
+                            0.0005639613373205066,
+                            0.000633670890238136,
+                            0.0008021970861591399,
+                            0.000590326264500618,
+                            0.0005889387102797627,
+                            0.0008753223228268325,
+                            0.0005973065854050219,
+                            0.0006219379720278084,
+                            0.00057617062702775,
+                            0.0005207609501667321,
+                            0.0008466871222481132,
+                            0.0006188637344166636,
+                            0.0010036476887762547,
+                            0.0006798261892981827,
+                            0.0005375382024794817,
+                            0.0006590557750314474,
+                            0.0006184886442497373,
+                            0.0010289946803823113,
+                            0.0005485789733938873,
+                            0.0007910518907010555,
+                            0.0006969828391447663,
+                            0.0006235119071789086,
+                            0.0006434324895963073,
+                            0.0005609303480014205,
+                            0.0006882688030600548,
+                            0.0007309198845177889,
+                            0.0010038648033514619,
+                            0.0005990704521536827,
+                            0.0006120153702795506,
+                            0.0005791003350168467,
+                            0.0005661892355419695,
+                            0.00059274781960994,
+                            0.0005696173175238073,
+                            0.0005811165319755673,
+                            0.0007125098491087556,
+                            0.000533096375875175,
+                            0.0008269093232229352,
+                            0.0005988495540805161,
+                            0.0005105560412630439,
+                            0.0005283320206217468,
+                            0.0007154869381338358,
+                            0.0005778921768069267,
+                            0.0006611645803786814,
+                            0.0006015953258611262,
+                            0.0005453621270135045,
+                            0.0007242046413011849,
+                            0.000603462802246213,
+                            0.0006433146772906184,
+                            0.0006820206181146204,
+                            0.0006770623149350286,
+                            0.0006002107402309775,
+                            0.0008109190384857357,
+                            0.0006165884551592171,
+                            0.0008948817267082632,
+                            0.0005931737832725048,
+                            0.0006163412472233176,
+                            0.0006722016842104495,
+                            0.0005597152630798519,
+                            0.0006282016402110457,
+                            0.0005553472437895834,
+                            0.0006037525599822402,
+                            0.00056364422198385,
+                            0.0006497236317954957,
+                            0.0008277156157419086,
+                            0.000786632823292166,
+                            0.0007973075262270868,
+                            0.0005336141330190003,
+                            0.0005098393885418773,
+                            0.0004903851659037173,
+                            0.000708080711774528,
+                            0.0006029171636328101,
+                            0.0006611855933442712,
+                            0.0005749794072471559,
+                            0.0005547298351302743,
+                            0.0005788862472400069,
+                            0.0005594221875071526,
+                            0.000546312949154526,
+                            0.0006466721533797681,
+                            0.0006392329814843833,
+                            0.0007881231722421944,
+                            0.0005623508477583528,
+                            0.0009592002024874091,
+                            0.0009472041856497526,
+                            0.0005608555511571467,
+                            0.0005217785364948213,
+                            0.000628713401965797,
+                            0.0006207749829627573,
+                            0.000610717514064163,
+                            0.0005290849949233234,
+                            0.0007344613550230861,
+                            0.0005630665109492838,
+                            0.0006058509461581707,
+                            0.0005683136405423284,
+                            0.0005793964955955744,
+                            0.0006603311048820615,
+                            0.0004887061077170074,
+                            0.0006031161174178123,
+                            0.0006421050056815147,
+                            0.0005450228927657008,
+                            0.0005967987817712128,
+                            0.000513144419528544,
+                            0.0007884746300987899,
+                            0.0005830653244629502,
+                            0.0006281277164816856,
+                            0.0005579506396315992,
+                            0.0005524722510017455,
+                            0.0005682780756615102,
+                            0.000983989448286593,
+                            0.0006211059517227113,
+                            0.0006051637465134263,
+                            0.0007320375880226493,
+                            0.0005538735422305763,
+                            0.0008806608384475112,
+                            0.0005652925465255976,
+                            0.0005584594327956438,
+                            0.0006967107183299959,
+                            0.0006555594736710191,
+                            0.0005792375304736197,
+                            0.000510142941493541,
+                            0.0005754318553954363,
+                            0.0006140467594377697,
+                            0.0005361258517950773,
+                            0.0005447514122352004,
+                            0.0005403526592999697,
+                            0.0006238463683985174,
+                            0.0006557085434906185,
+                            0.000626861525233835,
+                            0.0005968212499283254
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 69,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "8": {
+                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type_is_module": false,
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 70,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.01608937606215477
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    },
+                    {
+                        "id": 67,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.03753050044178963
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 71,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8"
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {
+            "0": {
+                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8"
+                    },
+                    {
+                        "id": 61,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 62,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ]
+            },
+            "1": {
+                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8"
+                    },
+                    {
+                        "id": 65,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 66,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ]
+            },
+            "2": {
+                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
+                "fqn": "inter_arch",
+                "input_tensor_infos": [
+                    {
+                        "id": 59,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8"
+                    },
+                    {
+                        "id": 69,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 70,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ]
+            }
+        },
+        "layer_output_infos": [
+            {
+                "id": 71,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "inter_arch:V_linears": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": []
+    },
+    "inter_arch:W_linears": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": []
+    },
+    "over_arch": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 80,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.float32"
+            }
+        ]
+    },
+    "over_arch:model": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "over_arch.model.1",
+                "input_tensor_infos": [
+                    {
+                        "id": 79,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.04151982069015503
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0007622003322467208
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 80,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32"
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 80,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.float32"
+            }
+        ]
+    },
+    "over_arch:model:0": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 79,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "over_arch:model:0:_mlp": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 79,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "over_arch:model:0:_mlp:0": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "over_arch.model.0._mlp.0._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 71,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.03963257744908333
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0006524756317958236,
+                            0.0005830027512274683,
+                            0.0005328580155037344,
+                            0.0005937081878073514,
+                            0.0006612135912291706,
+                            0.0005683698691427708,
+                            0.0006676965858787298,
+                            0.0008024597191251814,
+                            0.0005250104004517198,
+                            0.0012525822967290878,
+                            0.0007042608922347426,
+                            0.0006970649701543152,
+                            0.0008275345899164677,
+                            0.0006589460535906255,
+                            0.0009608828695490956,
+                            0.0005618117866106331,
+                            0.0005433103651739657,
+                            0.0010559102520346642,
+                            0.0010643166024237871,
+                            0.0007940800278447568,
+                            0.0005300909397192299,
+                            0.00045231598778627813,
+                            0.001031958614476025,
+                            0.000665140338242054,
+                            0.0006561845075339079,
+                            0.000767504156101495,
+                            0.0010487537365406752,
+                            0.0009493622928857803,
+                            0.0004503098607528955,
+                            0.0005853957845829427,
+                            0.0005926244775764644,
+                            0.0006097463774494827,
+                            0.0010205834405496716,
+                            0.0014860552037134767,
+                            0.0012734555639326572,
+                            0.0011275106808170676,
+                            0.0006563041242770851,
+                            0.0005434985505416989,
+                            0.0007068745326250792,
+                            0.0007832504343241453,
+                            0.0006326151778921485,
+                            0.0006725302082486451,
+                            0.0007715582614764571,
+                            0.0007140066591091454,
+                            0.0006247721030376852,
+                            0.0009152868879027665,
+                            0.0007446494419127703,
+                            0.0005306339007802308,
+                            0.0006052838289178908,
+                            0.0006119462195783854,
+                            0.0011588088236749172,
+                            0.0005619250587187707,
+                            0.000652515736874193,
+                            0.0005541375721804798,
+                            0.0006329460884444416,
+                            0.0005745276575908065,
+                            0.000824779155664146,
+                            0.0005778567865490913,
+                            0.0006792581407353282,
+                            0.0005817872006446123,
+                            0.0012376646045595407,
+                            0.0005263470811769366,
+                            0.0006723012775182724,
+                            0.0008856904460117221,
+                            0.0007506019319407642,
+                            0.0006921171443536878,
+                            0.000506663927808404,
+                            0.0006372245261445642,
+                            0.0007818072335794568,
+                            0.0005996691761538386,
+                            0.0007046354003250599,
+                            0.0007302319281734526,
+                            0.0005861080717295408,
+                            0.0007806236972101033,
+                            0.0006355338264256716,
+                            0.00069527787854895,
+                            0.0009608413092792034,
+                            0.0008264210191555321,
+                            0.0005999411223456264,
+                            0.0004038240003865212,
+                            0.000590808573178947,
+                            0.0007318562711589038,
+                            0.000749179336708039,
+                            0.0005344580276869237,
+                            0.0007254822412505746,
+                            0.0006098814774304628,
+                            0.0006267998833209276,
+                            0.0009896305855363607,
+                            0.0005726031959056854,
+                            0.000713455374352634,
+                            0.0006885855691507459,
+                            0.0006532000843435526,
+                            0.0006689404835924506,
+                            0.0006660834769718349,
+                            0.0006241864175535738,
+                            0.0006505982601083815,
+                            0.0005939678521826863,
+                            0.0008622792083770037,
+                            0.0007486206013709307,
+                            0.000687137246131897,
+                            0.001620420953258872,
+                            0.0005408972501754761,
+                            0.0012286958517506719,
+                            0.0007593865739181638,
+                            0.0008089187322184443,
+                            0.0009904456092044711,
+                            0.0006416584947146475,
+                            0.0008374062017537653,
+                            0.0008210075320675969,
+                            0.000767313817050308,
+                            0.0005508728208951652,
+                            0.0006707045249640942,
+                            0.0006045702612027526,
+                            0.0006886309129185975,
+                            0.0005901749827899039,
+                            0.0006030809599906206,
+                            0.0006524524069391191,
+                            0.0010539358481764793,
+                            0.0006129087996669114,
+                            0.0008696469012647867,
+                            0.0007082120282575488,
+                            0.0014049746096134186,
+                            0.000645416381303221,
+                            0.0005156938568688929,
+                            0.0004947902634739876,
+                            0.0006774991052225232,
+                            0.000971378234680742,
+                            0.0007075003813952208,
+                            0.0008161476580426097,
+                            0.0013682624557986856,
+                            0.0005673284758813679,
+                            0.000676686642691493,
+                            0.0017991267377510667,
+                            0.0009919072035700083,
+                            0.0006394893862307072,
+                            0.0007926374673843384,
+                            0.0007246306049637496,
+                            0.0009251240408048034,
+                            0.0009770557517185807,
+                            0.0007387450896203518,
+                            0.0014700844185426831,
+                            0.0009135324507951736,
+                            0.0007888193940743804,
+                            0.0006198180490173399,
+                            0.0008092082571238279,
+                            0.000647401437163353,
+                            0.0008085803128778934,
+                            0.001186860608868301,
+                            0.000646728731226176,
+                            0.000737213937100023,
+                            0.0006398428231477737,
+                            0.000718064489774406,
+                            0.0006077733705751598,
+                            0.0006149801192805171,
+                            0.001357105909846723,
+                            0.0008891353500075638,
+                            0.0009040357545018196,
+                            0.0005779667408205569,
+                            0.0006541495095007122,
+                            0.0006593502475880086,
+                            0.0011494321515783668,
+                            0.0008229032391682267,
+                            0.000540935609024018,
+                            0.0010768395150080323,
+                            0.0006562250200659037,
+                            0.0006441249861381948,
+                            0.0006463474710471928,
+                            0.0010264052543789148,
+                            0.000969059590715915,
+                            0.001897352747619152,
+                            0.0009705731063149869,
+                            0.0005479457904584706,
+                            0.0007067191181704402,
+                            0.0007894833688624203,
+                            0.000604870670940727,
+                            0.0016852535773068666,
+                            0.0007504631066694856,
+                            0.0006901359301991761,
+                            0.0005422300891950727,
+                            0.0005684404750354588,
+                            0.0006684219697490335,
+                            0.0005576451076194644,
+                            0.0005162637098692358,
+                            0.0005255104624666274,
+                            0.0008227318176068366,
+                            0.0008867628639563918,
+                            0.001002840232104063,
+                            0.001167431939393282,
+                            0.0009770909091457725,
+                            0.0006378170219250023,
+                            0.0006709573790431023,
+                            0.001258184784092009,
+                            0.0016726505709812045,
+                            0.0006308098090812564,
+                            0.0006683635292574763,
+                            0.0004168228479102254,
+                            0.0007023995858617127,
+                            0.001328894984908402,
+                            0.0007430414552800357,
+                            0.000738993810955435,
+                            0.0006891862722113729,
+                            0.0005112405051477253,
+                            0.0008361341897398233,
+                            0.0005878934171050787,
+                            0.000672191905323416,
+                            0.0006971266702748835,
+                            0.0009516200516372919,
+                            0.0006750481552444398,
+                            0.0012501354794949293,
+                            0.0007360093877650797,
+                            0.0008408424910157919,
+                            0.0006230273866094649,
+                            0.000618400052189827,
+                            0.000694845337420702,
+                            0.0006644437671639025,
+                            0.0005307486862875521,
+                            0.0006387433386407793,
+                            0.0006404966698028147,
+                            0.0007669288315810263,
+                            0.0006691318121738732,
+                            0.0005561733269132674,
+                            0.0005342485965229571,
+                            0.0008629201911389828,
+                            0.0006823228322900832,
+                            0.0006400076672434807,
+                            0.0005892260815016925,
+                            0.0016322049777954817,
+                            0.0006629402632825077,
+                            0.000542009191121906,
+                            0.0012109210947528481,
+                            0.0006654296885244548,
+                            0.000630803348030895,
+                            0.0005117054679431021,
+                            0.000600938277784735,
+                            0.0007397328736260533,
+                            0.000865007983520627,
+                            0.0005533203366212547,
+                            0.0005843895487487316,
+                            0.0012111107353121042,
+                            0.000715410802513361,
+                            0.000556318846065551,
+                            0.000796379754319787,
+                            0.0006064907065592706,
+                            0.0013348519569262862,
+                            0.0006024658214300871,
+                            0.0008236340363509953,
+                            0.0007649407489225268,
+                            0.0006043059984222054,
+                            0.0009718523360788822,
+                            0.0006345193833112717,
+                            0.0009985349606722593,
+                            0.0007417187443934381,
+                            0.0014045732095837593,
+                            0.0005761345964856446,
+                            0.0007510728319175541,
+                            0.000708447361830622,
+                            0.0005494263023138046,
+                            0.000414042646298185,
+                            0.0005621177842840552,
+                            0.0005934873479418457,
+                            0.0006163967191241682,
+                            0.0004890726995654404,
+                            0.0012738098157569766,
+                            0.0008057748782448471,
+                            0.0006453674868680537,
+                            0.0008247778750956059,
+                            0.0006085478817112744,
+                            0.0008311715209856629,
+                            0.0014232872053980827,
+                            0.0010768562788143754,
+                            0.0006527877994813025,
+                            0.001342790201306343,
+                            0.0008041166584007442,
+                            0.0007310926448553801,
+                            0.0006161658675409853,
+                            0.001278816838748753,
+                            0.0006079203449189663,
+                            0.0009844563901424408,
+                            0.0006827431498095393,
+                            0.000959579658228904,
+                            0.0006245423574000597,
+                            0.0005169829237274826,
+                            0.0007256833487190306,
+                            0.001252972288057208,
+                            0.0006664976826868951,
+                            0.0011853516334667802,
+                            0.0008067220915108919,
+                            0.0006806396413594484,
+                            0.0005189778748899698,
+                            0.001154152094386518,
+                            0.0016463745851069689,
+                            0.0007296705152839422,
+                            0.0004966111155226827,
+                            0.0005802034866064787,
+                            0.0014263682533055544,
+                            0.00108760641887784,
+                            0.0005785427056252956,
+                            0.0004752358654513955,
+                            0.0009489961666986346,
+                            0.0008068829192779958,
+                            0.0009285283158533275,
+                            0.0008509764447808266,
+                            0.0008649718365631998,
+                            0.0006978833698667586,
+                            0.0006279246881604195,
+                            0.0011238137958571315,
+                            0.0005839085788466036,
+                            0.0007487628608942032,
+                            0.0006132452981546521,
+                            0.0005693636485375464,
+                            0.001035313936881721,
+                            0.000656121876090765,
+                            0.0006261906237341464,
+                            0.0016192039474844933,
+                            0.0005611933302134275,
+                            0.0005230784881860018,
+                            0.0006303816335275769,
+                            0.0011645981576293707,
+                            0.0012850045459344983,
+                            0.0006710198940709233,
+                            0.0005317329196259379,
+                            0.0007282670121639967,
+                            0.0005409839795902371,
+                            0.0005553693627007306,
+                            0.0005989729543216527,
+                            0.0006533776759169996,
+                            0.0006881059380248189,
+                            0.0012559023452922702,
+                            0.0005750397685915232,
+                            0.001279589137993753,
+                            0.0007285907049663365,
+                            0.0012623395305126905,
+                            0.0004474413872230798,
+                            0.0006686791311949492,
+                            0.0006621580105274916,
+                            0.0006775293149985373,
+                            0.0005252978880889714,
+                            0.0006864758324809372,
+                            0.0007465987582691014,
+                            0.0006730711902491748,
+                            0.000798654044046998,
+                            0.0005705353105440736,
+                            0.0006016611005179584,
+                            0.0005722312489524484,
+                            0.000639223144389689,
+                            0.0006624904926866293,
+                            0.0015691135777160525,
+                            0.0006503246841020882,
+                            0.0010688946349546313,
+                            0.001170934410765767,
+                            0.0005997002008371055,
+                            0.0005447817966341972,
+                            0.0012661967193707824,
+                            0.0009861881844699383,
+                            0.0006367606110870838,
+                            0.0005618043360300362,
+                            0.0006064396002329886,
+                            0.0014662696048617363,
+                            0.0006311855395324528,
+                            0.000578194682020694,
+                            0.0008195602567866445,
+                            0.0005558765260502696,
+                            0.0011730864644050598,
+                            0.0005988992052152753,
+                            0.00052532838890329,
+                            0.0006506553036160767,
+                            0.0008235081913881004,
+                            0.0007908449042588472,
+                            0.0008432971080765128,
+                            0.0012706476263701916,
+                            0.0005929248291067779,
+                            0.0006950330571271479,
+                            0.0007016637828201056,
+                            0.0007224761648103595,
+                            0.0007029881817288697,
+                            0.0005396137712523341,
+                            0.0007692644139751792,
+                            0.0005448743468150496,
+                            0.0006803314900025725,
+                            0.0009041817393153906,
+                            0.0007612495683133602,
+                            0.0007954597822390497,
+                            0.000763938354793936,
+                            0.00044284487375989556,
+                            0.001395230763591826,
+                            0.0005942063289694488,
+                            0.0006007755873724818,
+                            0.0005244857165962458,
+                            0.0007656614761799574,
+                            0.0009027895284816623,
+                            0.0007396965520456433,
+                            0.0011739114997908473,
+                            0.0006069990922696888,
+                            0.0006182430661283433,
+                            0.0007382381008937955,
+                            0.0008098558755591512,
+                            0.0009702307870611548,
+                            0.0011961141135543585,
+                            0.0007279645069502294,
+                            0.0011475131614133716,
+                            0.0005214876146055758,
+                            0.0016241009579971433,
+                            0.001001782133243978,
+                            0.0006782648852095008,
+                            0.001193162053823471,
+                            0.0007628849125467241,
+                            0.0008343815570697188,
+                            0.0005801442894153297,
+                            0.0007014219299890101,
+                            0.0007195632206276059,
+                            0.0007329740910790861,
+                            0.0013682267162948847,
+                            0.0008865426643751562,
+                            0.0010245096636936069,
+                            0.0009985582437366247,
+                            0.0005685520591214299,
+                            0.0007716232794336975,
+                            0.000563573557883501,
+                            0.0008810853469185531,
+                            0.000634254771284759,
+                            0.000699271506164223,
+                            0.0013130102306604385,
+                            0.0007598600932396948,
+                            0.0008229886880144477,
+                            0.0007153492188081145,
+                            0.0004872058634646237,
+                            0.0006044415640644729,
+                            0.00048524021985940635,
+                            0.000594704004470259,
+                            0.0013038618490099907,
+                            0.0006212126463651657,
+                            0.0008205929188989103,
+                            0.0010290987556800246,
+                            0.0005835560150444508,
+                            0.0007094572065398097,
+                            0.0010461630299687386,
+                            0.0008886082796379924,
+                            0.0007824527565389872,
+                            0.0008179688011296093,
+                            0.0006885174079798162,
+                            0.0009109302191063762,
+                            0.0006978776655159891,
+                            0.0006608805852010846,
+                            0.0011151155922561884,
+                            0.0006641866057179868,
+                            0.0008335679303854704,
+                            0.001226708642207086,
+                            0.0010822738986462355,
+                            0.001215369557030499,
+                            0.0008342767832800746,
+                            0.0006089520757086575,
+                            0.0008319059852510691,
+                            0.00108604715205729,
+                            0.0005628687213174999,
+                            0.0005562207661569118,
+                            0.0010228322353214025,
+                            0.0008859948720782995,
+                            0.0008337029139511287,
+                            0.0006537845474667847,
+                            0.0008466040599159896,
+                            0.0007119046640582383,
+                            0.0006104364874772727,
+                            0.0006544673815369606,
+                            0.0006664818502031267,
+                            0.0007010228582657874,
+                            0.0007094722823239863,
+                            0.0011393040185794234,
+                            0.0012534792767837644,
+                            0.0004690915229730308,
+                            0.0007427737582474947,
+                            0.0010785538470372558,
+                            0.0008124964660964906,
+                            0.0008309290860779583,
+                            0.0005443574045784771,
+                            0.0006756943184882402,
+                            0.0006038204301148653,
+                            0.0006380255217663944,
+                            0.0005390677833929658,
+                            0.0007413743296638131,
+                            0.0005906802252866328,
+                            0.0005397173226810992,
+                            0.0005776295438408852,
+                            0.0006055396515876055,
+                            0.001173705910332501,
+                            0.000984863960184157,
+                            0.0005698304739780724,
+                            0.0005976701504550874,
+                            0.0006889742217026651,
+                            0.0008824350661598146,
+                            0.0007669352344237268,
+                            0.0006478721043094993,
+                            0.0008456395589746535,
+                            0.0008933895151130855,
+                            0.0009811160853132606,
+                            0.0005001409444957972,
+                            0.0007260318379849195,
+                            0.0005765112582594156,
+                            0.00058128556702286,
+                            0.0010118978098034859,
+                            0.0008525068988092244,
+                            0.0008656661375425756,
+                            0.0009582648053765297,
+                            0.0009095269488170743,
+                            0.0008212847169488668,
+                            0.0006527090445160866,
+                            0.00052688701543957,
+                            0.0005282641504891217,
+                            0.0005831000744365156,
+                            0.000708655163180083,
+                            0.000879493891261518,
+                            0.0006397967808879912,
+                            0.0006545981741510332,
+                            0.0007385017233900726,
+                            0.0006436569965444505,
+                            0.0008645160123705864,
+                            0.0011941406410187483,
+                            0.0004914570599794388,
+                            0.0005904400022700429,
+                            0.0006936010322533548,
+                            0.0005110740894451737,
+                            0.0007272091461345553,
+                            0.0006062098545953631,
+                            0.000515967549290508,
+                            0.0006554198917001486,
+                            0.0006061721360310912,
+                            0.0007207796443253756,
+                            0.0009388610487803817,
+                            0.0007894348818808794,
+                            0.0009642571094445884,
+                            0.0005278083845041692,
+                            0.0005395899643190205,
+                            0.0007439987966790795,
+                            0.0006118636811152101,
+                            0.0006976831937208772,
+                            0.0006397608085535467,
+                            0.000690947228576988,
+                            0.0006779690738767385,
+                            0.0006774225039407611,
+                            0.0005673198611475527,
+                            0.0006491863750852644,
+                            0.000581262051127851,
+                            0.0008383641834370792,
+                            0.00117177062202245,
+                            0.000602753774728626,
+                            0.0006406339816749096,
+                            0.0005072711501270533,
+                            0.00047014051233418286,
+                            0.00044659795821644366,
+                            0.0007701131980866194,
+                            0.0008233151165768504,
+                            0.0005277334712445736,
+                            0.0007534264004789293,
+                            0.0006634113960899413,
+                            0.0008647429058328271,
+                            0.0005124252638779581,
+                            0.0006749372114427388,
+                            0.0006448228377848864,
+                            0.0005601090379059315,
+                            0.0006865259492769837,
+                            0.0008302323403768241,
+                            0.0010373146506026387,
+                            0.0008844147669151425,
+                            0.0007673210930079222,
+                            0.0004886587848886847,
+                            0.0004962943494319916,
+                            0.0013126564444974065,
+                            0.0010737006086856127,
+                            0.0006696421187371016,
+                            0.0006178843323141336,
+                            0.000607631984166801,
+                            0.0005419758963398635,
+                            0.000710859545506537,
+                            0.0006018277490511537,
+                            0.0007471634889952838,
+                            0.0008029809105210006,
+                            0.0007198998355306685,
+                            0.0005988481570966542,
+                            0.0009661775548011065,
+                            0.0007078018388710916,
+                            0.0007000485784374177,
+                            0.0008112714858725667,
+                            0.0006677646888419986,
+                            0.0006162812351249158,
+                            0.0007040456985123456,
+                            0.0009912133682519197,
+                            0.0006095203571021557,
+                            0.0007373302360065281,
+                            0.0009651418076828122,
+                            0.0008452601032331586,
+                            0.0007864505751058459,
+                            0.0007301364676095545,
+                            0.0008134028175845742,
+                            0.0009087399812415242,
+                            0.0006471886881627142,
+                            0.000620750361122191,
+                            0.0006738413940183818,
+                            0.0008588361088186502,
+                            0.0006089923554100096,
+                            0.0007942375377751887,
+                            0.0006658442434854805,
+                            0.0008694135467521846,
+                            0.0009383915457874537,
+                            0.00047031333087943494,
+                            0.0006519275484606624,
+                            0.0015533070545643568,
+                            0.0007110235746949911,
+                            0.000596866593696177,
+                            0.0009275038028135896,
+                            0.0011968291364610195,
+                            0.0006084782653488219,
+                            0.0008557692053727806,
+                            0.00054791325237602,
+                            0.0008198049035854638,
+                            0.0005787248956039548,
+                            0.0007824009517207742,
+                            0.0013026088709011674,
+                            0.001204103697091341,
+                            0.000666916836053133,
+                            0.0012014053063467145,
+                            0.001075370004400611,
+                            0.000618891092017293,
+                            0.00047439063200727105,
+                            0.0006848077173344791,
+                            0.0005811586743220687,
+                            0.0006995286676101387,
+                            0.001583934179507196,
+                            0.0005193555844016373,
+                            0.0009656983893364668,
+                            0.0005685413489118218,
+                            0.0005368883721530437,
+                            0.0005690444377250969,
+                            0.0010896532330662012,
+                            0.0008998399716801941,
+                            0.0005715474835596979,
+                            0.001135902013629675,
+                            0.000683841819409281,
+                            0.0015482025919482112,
+                            0.0006698979996144772,
+                            0.0005719265318475664,
+                            0.0005981206195428967,
+                            0.0005931049236096442,
+                            0.0005965330637991428,
+                            0.0006821951828896999,
+                            0.0014198649441823363,
+                            0.0007854808354750276,
+                            0.0005620442098006606,
+                            0.000715198228135705,
+                            0.000678872806020081,
+                            0.0005413230392150581,
+                            0.0005383276729844511,
+                            0.0009201165521517396,
+                            0.000648867164272815,
+                            0.0011457609944045544,
+                            0.0006391564384102821,
+                            0.000533323734998703,
+                            0.0005945664015598595,
+                            0.0006568116368725896,
+                            0.001418829197064042,
+                            0.0007951294537633657,
+                            0.0006048528011888266,
+                            0.0005724955117329955,
+                            0.000727491918951273,
+                            0.0009613253059796989,
+                            0.0005400183727033436,
+                            0.000706421968061477,
+                            0.0005649305530823767,
+                            0.0006636874168179929,
+                            0.0008539884001947939,
+                            0.0007648233440704644,
+                            0.0006250097067095339,
+                            0.001350661856122315,
+                            0.0006656144396401942,
+                            0.0008363249362446368,
+                            0.0005720870103687048,
+                            0.0007848450331948698,
+                            0.0012050856603309512,
+                            0.0006789648905396461,
+                            0.0005399481160566211,
+                            0.0006369933835230768,
+                            0.0007509341812692583,
+                            0.0009613109868951142,
+                            0.0008885667775757611,
+                            0.0006174836889840662,
+                            0.000705633545294404,
+                            0.0005672239931300282,
+                            0.0005709343240596354,
+                            0.0007202911074273288,
+                            0.0006294110789895058,
+                            0.0009319105884060264,
+                            0.0007067770347930491,
+                            0.0011589163914322853,
+                            0.0008147672051563859,
+                            0.000810261582955718,
+                            0.00046063255285844207,
+                            0.0005786246038042009,
+                            0.0005881876568309963,
+                            0.000814545142930001,
+                            0.0008576664258725941,
+                            0.0014480475801974535,
+                            0.000684817845467478,
+                            0.000685538281686604,
+                            0.0007262881263159215,
+                            0.0006014537066221237,
+                            0.0006594664882868528,
+                            0.0005542123108170927,
+                            0.0006595243467018008,
+                            0.0007399325841106474,
+                            0.0006288132863119245,
+                            0.0011309698456898332,
+                            0.0006201270734891295,
+                            0.0006123575149103999,
+                            0.000607012421824038,
+                            0.0006381271523423493,
+                            0.0012307383585721254,
+                            0.0007419998291879892,
+                            0.0007119080401025712,
+                            0.0006362175918184221,
+                            0.0009344108984805644,
+                            0.000611891271546483,
+                            0.0006295218481682241,
+                            0.0005824012914672494,
+                            0.0011731303529813886,
+                            0.0006737064104527235,
+                            0.0006326079019345343,
+                            0.0006597090978175402,
+                            0.0004332766984589398,
+                            0.0008711983682587743,
+                            0.0011107047321274877,
+                            0.0011588949710130692,
+                            0.000672138121444732,
+                            0.001215190626680851,
+                            0.0011990801431238651,
+                            0.000881969986949116,
+                            0.0005186385242268443,
+                            0.0006933835102245212,
+                            0.0004938743077218533,
+                            0.0005319016054272652,
+                            0.0006543559720739722,
+                            0.0004919477505609393,
+                            0.001286143553443253,
+                            0.0010097264312207699,
+                            0.0005679705063812435,
+                            0.0006606851238757372,
+                            0.0004679433477576822,
+                            0.0006834640516899526,
+                            0.0003490756789688021,
+                            0.0006554378778673708,
+                            0.0009800883708521724,
+                            0.0017448380822315812,
+                            0.0005888597806915641,
+                            0.0006106723449192941,
+                            0.0006519539165310562,
+                            0.0008018246735446155,
+                            0.0005634083645418286,
+                            0.0007861882913857698,
+                            0.0007564613479189575,
+                            0.0008253294508904219,
+                            0.0008321021450683475,
+                            0.0007896038005128503,
+                            0.0005516746896319091,
+                            0.0006926748319528997,
+                            0.000683494727127254,
+                            0.000661246245726943,
+                            0.0013015249278396368,
+                            0.0012562476331368089,
+                            0.0012083452893421054,
+                            0.0014969364274293184,
+                            0.0005777698825113475,
+                            0.0011726923985406756,
+                            0.0012562418123707175,
+                            0.00046190517605282366,
+                            0.0009937373688444495,
+                            0.000775026623159647,
+                            0.0012327489675953984,
+                            0.0005954226362518966,
+                            0.0011185468174517155,
+                            0.0016989901196211576,
+                            0.000564415764529258,
+                            0.0006311562610790133,
+                            0.0009466158808209002,
+                            0.0009001410217024386,
+                            0.0007511824369430542,
+                            0.0006797880050726235,
+                            0.0007659794064238667,
+                            0.0006914565456099808,
+                            0.0011699805036187172,
+                            0.001141633139923215,
+                            0.0007956832414492965,
+                            0.000544023874681443,
+                            0.0006454846588894725,
+                            0.0006897478015162051,
+                            0.0005392273887991905,
+                            0.0006027352064847946,
+                            0.0010556158376857638,
+                            0.00043817938421852887,
+                            0.0013528201961889863,
+                            0.0010557830100879073,
+                            0.0006414923118427396,
+                            0.0008660315070301294,
+                            0.0006415839889086783,
+                            0.0005576511612161994,
+                            0.001209845533594489,
+                            0.0006432805093936622,
+                            0.0009104650234803557,
+                            0.0011839618673548102,
+                            0.000692327506840229,
+                            0.0006386519526131451,
+                            0.0007390736718662083,
+                            0.0016133026219904423,
+                            0.0007941718795336783,
+                            0.0006401382270269096,
+                            0.0009923892794176936,
+                            0.0006844333256594837,
+                            0.001223024562932551,
+                            0.0008679967722855508,
+                            0.001596940099261701,
+                            0.0006907226634211838,
+                            0.0012179643381386995,
+                            0.0011453625047579408,
+                            0.0007270281785167754,
+                            0.0005954531952738762,
+                            0.00046905496856197715,
+                            0.0007979062502272427,
+                            0.0003911541716661304,
+                            0.0007825930370017886,
+                            0.0012180806370452046,
+                            0.00106238410808146,
+                            0.0014883498661220074,
+                            0.0010852975538000464,
+                            0.0006212134612724185,
+                            0.0007391534163616598,
+                            0.00073702895315364,
+                            0.0010130808223038912,
+                            0.0007469221018254757,
+                            0.0006276454660110176,
+                            0.0005134720122441649,
+                            0.0007043119403533638,
+                            0.0006824063020758331,
+                            0.0006637214682996273,
+                            0.0007526191766373813,
+                            0.000846075767185539,
+                            0.0006863265298306942,
+                            0.0008026761934161186,
+                            0.0012956243008375168,
+                            0.0009172737482003868,
+                            0.0008007778087630868,
+                            0.0006753631751053035,
+                            0.0006253708270378411,
+                            0.0006575853331014514,
+                            0.0006481084274128079,
+                            0.0005261931801214814,
+                            0.0005640859599225223,
+                            0.0007180894026532769,
+                            0.0005630103987641633,
+                            0.0008500644471496344,
+                            0.0005712871789000928,
+                            0.0006573980790562928,
+                            0.0006503087934106588,
+                            0.0005966360913589597,
+                            0.0008427710854448378,
+                            0.0007230520714074373,
+                            0.0006481027812696993,
+                            0.0005906702135689557,
+                            0.0006604077061638236,
+                            0.0006547679076902568,
+                            0.0005271546542644501,
+                            0.0006760006654076278,
+                            0.0008970536291599274,
+                            0.0007924412493593991,
+                            0.000706030405126512,
+                            0.0013736693654209375,
+                            0.0005960756679996848,
+                            0.0006004491588100791,
+                            0.0006854602252133191,
+                            0.0008137762197293341,
+                            0.0007738486165180802,
+                            0.0005627921200357378,
+                            0.0008861810201779008,
+                            0.0005573134403675795,
+                            0.0005988131742924452,
+                            0.0005256404401734471,
+                            0.0006381064304150641,
+                            0.0006617616745643318,
+                            0.00041274065733887255,
+                            0.0008215995039790869,
+                            0.0006105442880652845,
+                            0.0007389284437522292,
+                            0.0006847382755950093,
+                            0.0006423346349038184,
+                            0.0006709294975735247,
+                            0.0011979348491877317,
+                            0.001762045081704855,
+                            0.0009851864306256175,
+                            0.0008361043874174356,
+                            0.0004362437466625124,
+                            0.0011576098622754216,
+                            0.0009099425515159965,
+                            0.0005281969206407666,
+                            0.000962300633545965,
+                            0.0007474017911590636,
+                            0.0006369519978761673,
+                            0.0005404900293797255,
+                            0.0006506743957288563,
+                            0.0005127650802023709,
+                            0.0005721734487451613,
+                            0.0010520420037209988,
+                            0.001272706314921379,
+                            0.00044300814624875784,
+                            0.0008762690122239292,
+                            0.000786739808972925,
+                            0.000887810718268156,
+                            0.0005210613016970456,
+                            0.0006736130453646183,
+                            0.0005252209957689047,
+                            0.0007015035371296108,
+                            0.001463692169636488,
+                            0.0005956862005405128,
+                            0.0005281604244373739,
+                            0.0006051120581105351,
+                            0.0007820960599929094,
+                            0.0007295802934095263,
+                            0.000716379436198622,
+                            0.0007459917687810957,
+                            0.0007694982923567295,
+                            0.0013484688242897391,
+                            0.0011137097608298063,
+                            0.0006131411646492779,
+                            0.0008004956762306392,
+                            0.00178135943133384,
+                            0.0006869127391837537,
+                            0.0009870331268757582,
+                            0.0006180040654726326,
+                            0.0006339642568491399,
+                            0.000672773050609976,
+                            0.0007207061862573028,
+                            0.0007271053036674857,
+                            0.0008506916929036379,
+                            0.0005910858162678778,
+                            0.0008117678808048368,
+                            0.0007325740298256278,
+                            0.0006856707041151822,
+                            0.0006105947541072965,
+                            0.0014264166820794344,
+                            0.0007257099496200681,
+                            0.0005982874426990747,
+                            0.0006193433655425906,
+                            0.0006085532950237393,
+                            0.0008751646964810789,
+                            0.0005486233858391643,
+                            0.0006491964450106025,
+                            0.0013933383161202073,
+                            0.0006974780699238181,
+                            0.0005741019849665463,
+                            0.0006536865839734674,
+                            0.0007001931662671268,
+                            0.001224698033183813,
+                            0.0009694860782474279,
+                            0.0007572437170892954,
+                            0.0007189347525127232,
+                            0.000816378160379827,
+                            0.0011236126301810145,
+                            0.0006210939609445632,
+                            0.000633695162832737,
+                            0.000785508134867996,
+                            0.0009406181634403765,
+                            0.0005022605182603002,
+                            0.0009548970847390592,
+                            0.0009236870100721717,
+                            0.0007531936280429363,
+                            0.0007324842736124992,
+                            0.0007503704400733113,
+                            0.0008211199892684817,
+                            0.0005993007798679173,
+                            0.0007780871237628162,
+                            0.0005905234720557928,
+                            0.0004984202096238732,
+                            0.0006574935396201909,
+                            0.0005235598655417562,
+                            0.000472771906061098,
+                            0.000582084758207202,
+                            0.0006631047581322491,
+                            0.0008032619953155518,
+                            0.000588608963880688,
+                            0.0007687024190090597,
+                            0.0007000963087193668,
+                            0.0011868264991790056,
+                            0.001003430224955082,
+                            0.0016845252830535173,
+                            0.0011295466683804989,
+                            0.0006729778833687305,
+                            0.0010070267599076033,
+                            0.0006483279285021126,
+                            0.0005307309911586344,
+                            0.0008289113757200539,
+                            0.0007763456087559462,
+                            0.0009893154492601752,
+                            0.0013245170703157783,
+                            0.0006207305123098195,
+                            0.0015590292168781161,
+                            0.001267123268917203,
+                            0.0008795392932370305,
+                            0.0005775861791335046,
+                            0.0006107842782512307,
+                            0.0008043512934818864,
+                            0.0006885397015139461,
+                            0.0006770114414393902,
+                            0.0005985439638607204,
+                            0.0005014885682612658,
+                            0.001374068553559482,
+                            0.0005983454175293446,
+                            0.0006956349825486541,
+                            0.0005553168011829257,
+                            0.0009033168898895383,
+                            0.0005764508387073874,
+                            0.0006778539391234517,
+                            0.0016231744084507227,
+                            0.0006020783330313861,
+                            0.0006597147439606488,
+                            0.0008063513087108731,
+                            0.0005594678805209696,
+                            0.0004922171356156468,
+                            0.0005358438938856125,
+                            0.00094301049830392,
+                            0.0008220780291594565
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 72,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.023695606738328934
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "over_arch.model.0._mlp.0",
+                "input_tensor_infos": [
+                    {
+                        "id": 72,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.023695606738328934
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 73,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.024861177429556847
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 73,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.024861177429556847
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "over_arch:model:0:_mlp:1": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "over_arch.model.0._mlp.1._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 73,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.024861177429556847
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0003642178780864924,
+                            0.0005359913338907063,
+                            0.001132896519266069,
+                            0.00047985161654651165,
+                            0.0005585832404904068,
+                            0.0003527008811943233,
+                            0.0003808389010373503,
+                            0.0009976738365367055,
+                            0.00031155816395767033,
+                            0.00028412966639734805,
+                            0.00028005498461425304,
+                            0.0003673212486319244,
+                            0.0004749326326418668,
+                            0.0006435640389099717,
+                            0.0004706500330939889,
+                            0.0008000647067092359,
+                            0.0003858283453155309,
+                            0.00039480128907598555,
+                            0.00034536453313194215,
+                            0.00036025303415954113,
+                            0.0010496347676962614,
+                            0.00028321114950813353,
+                            0.0005664917407557368,
+                            0.00037216133205220103,
+                            0.0006849024211987853,
+                            0.00045106923789717257,
+                            0.00044631294440478086,
+                            0.001251291949301958,
+                            0.00030767786665819585,
+                            0.0008218698785640299,
+                            0.0003787438035942614,
+                            0.0005398737266659737,
+                            0.00033994289697147906,
+                            0.0011434510815888643,
+                            0.00039532757364213467,
+                            0.00083381705917418,
+                            0.0004943276289850473,
+                            0.0005752966972067952,
+                            0.00028279388789087534,
+                            0.0002829310833476484,
+                            0.0002836508792825043,
+                            0.000605847395490855,
+                            0.0005264958599582314,
+                            0.001067151315510273,
+                            0.0002836299827322364,
+                            0.0003070708771701902,
+                            0.0006424721796065569,
+                            0.00130570272449404,
+                            0.0006646810215897858,
+                            0.0003884789766743779,
+                            0.0003740566608030349,
+                            0.0003156714083161205,
+                            0.0003815868985839188,
+                            0.0003045675402972847,
+                            0.0002840614761225879,
+                            0.00030357984360307455,
+                            0.0005073168431408703,
+                            0.00027830866747535765,
+                            0.00028414936969056726,
+                            0.00043508681119419634,
+                            0.0002841142995748669,
+                            0.00037687490112148225,
+                            0.00028394663240760565,
+                            0.0005671985563822091,
+                            0.0004257132823113352,
+                            0.0006239438662305474,
+                            0.0003002723678946495,
+                            0.0004571422759909183,
+                            0.00037576412432827055,
+                            0.00040418212302029133,
+                            0.000504864496178925,
+                            0.00038745615165680647,
+                            0.00035763796768151224,
+                            0.00037946191150695086,
+                            0.0002832069294527173,
+                            0.00045827683061361313,
+                            0.0002831204328685999,
+                            0.0005512615316547453,
+                            0.00044848924153484404,
+                            0.00028363431920297444,
+                            0.0003583719953894615,
+                            0.0009461860172450542,
+                            0.0006753078778274357,
+                            0.0002931411436293274,
+                            0.0003133818390779197,
+                            0.00028294450021348894,
+                            0.0010372786782681942,
+                            0.0011928061721846461,
+                            0.00032249229843728244,
+                            0.00033460231497883797,
+                            0.0003879878204315901,
+                            0.0003822189464699477,
+                            0.0005665172939188778,
+                            0.0008397395722568035,
+                            0.0002826140553224832,
+                            0.0008442078251391649,
+                            0.0003489274822641164,
+                            0.0003884479228872806,
+                            0.0002888966992031783,
+                            0.0004019843472633511,
+                            0.0009316186769865453,
+                            0.00035116722574457526,
+                            0.00029697225545533,
+                            0.00029997614910826087,
+                            0.0003121806657873094,
+                            0.0005984760355204344,
+                            0.0004888762487098575,
+                            0.0005529408226720989,
+                            0.0005506021552719176,
+                            0.0007433249847963452,
+                            0.00034386449260637164,
+                            0.0005101020215079188,
+                            0.0008129319176077843,
+                            0.00033162839827127755,
+                            0.000458107388112694,
+                            0.000924109888728708,
+                            0.0004301180597394705,
+                            0.0009746929281391203,
+                            0.0006471707602031529,
+                            0.00030065170722082257,
+                            0.0007030424894765019,
+                            0.0006645089597441256,
+                            0.0005074728978797793,
+                            0.0002756863832473755,
+                            0.0003816004900727421,
+                            0.0003600970667321235,
+                            0.000479770329548046,
+                            0.001069256686605513,
+                            0.000643627776298672,
+                            0.0003269365115556866,
+                            0.00034666623105295,
+                            0.0002843642432708293,
+                            0.0012293420732021332,
+                            0.00030714034801349044,
+                            0.000702757213730365,
+                            0.00045393381151370704,
+                            0.0003897251153830439,
+                            0.0002840948582161218,
+                            0.0003086741198785603,
+                            0.00032426766119897366,
+                            0.00035009137354791164,
+                            0.00034788576886057854,
+                            0.00030576990684494376,
+                            0.0005068911705166101,
+                            0.00041063420940190554,
+                            0.00028365181060507894,
+                            0.0003086649812757969,
+                            0.0006062181782908738,
+                            0.0005956823006272316,
+                            0.00036453839857131243,
+                            0.00040132220601662993,
+                            0.0005488541210070252,
+                            0.001070832833647728,
+                            0.001278785988688469,
+                            0.0004051370488014072,
+                            0.0002910397306550294,
+                            0.0003067479992751032,
+                            0.0009990269318223,
+                            0.0003735458885785192,
+                            0.0006931772222742438,
+                            0.00039756501791998744,
+                            0.0004019938060082495,
+                            0.0004963026149198413,
+                            0.00042962937732227147,
+                            0.0005159345455467701,
+                            0.00028421691968105733,
+                            0.00032827709219418466,
+                            0.0003451900265645236,
+                            0.00036992429522797465,
+                            0.0003402381844352931,
+                            0.0005786828696727753,
+                            0.00032286482746712863,
+                            0.0007704262970946729,
+                            0.0004257761174812913,
+                            0.0010172964539378881,
+                            0.00043923757039010525,
+                            0.0003589391999412328,
+                            0.0003246353007853031,
+                            0.0006230596336536109,
+                            0.0010658347746357322,
+                            0.000388928281608969,
+                            0.0003042728058062494,
+                            0.0011717882007360458,
+                            0.00038682532613165677,
+                            0.0003040395095013082,
+                            0.0003069276863243431,
+                            0.0006418314296752214,
+                            0.0005030770553275943,
+                            0.0005897496594116092,
+                            0.0005917403032071888,
+                            0.00027636130107566714,
+                            0.0014055819483473897,
+                            0.00032722874311730266,
+                            0.0013733970699831843,
+                            0.00028682962874881923,
+                            0.0003060274466406554,
+                            0.0008947619935497642,
+                            0.00032874266617000103,
+                            0.0003177314065396786,
+                            0.00035868087434209883,
+                            0.0003248322755098343,
+                            0.00028383536846376956,
+                            0.000578440900426358,
+                            0.00034570571733638644,
+                            0.00029169689514674246,
+                            0.00028398545691743493,
+                            0.0007273133960552514,
+                            0.0002835833583958447,
+                            0.0012339780805632472,
+                            0.0011406937846913934,
+                            0.00028901471523568034,
+                            0.00032221528817899525,
+                            0.0003722345572896302,
+                            0.0004422707424964756,
+                            0.00043508209637366235,
+                            0.0003739533422049135,
+                            0.0007685390883125365,
+                            0.0005698943859897554,
+                            0.0005637376452796161,
+                            0.0014216185081750154,
+                            0.00030379026429727674,
+                            0.00036978928255848587,
+                            0.00036584274494089186,
+                            0.0005524404114112258,
+                            0.00044688486377708614,
+                            0.0002860029926523566,
+                            0.0004748975916299969,
+                            0.00032478675711899996,
+                            0.0002786073018796742,
+                            0.0004568768199533224,
+                            0.00031826927443034947,
+                            0.00036352197639644146,
+                            0.00028420015587471426,
+                            0.000342807499691844,
+                            0.0002840289380401373,
+                            0.0003962450136896223,
+                            0.00034581616637296975,
+                            0.00031299894908443093,
+                            0.0007026650127954781,
+                            0.0004428593965712935,
+                            0.00030382152181118727,
+                            0.0003477045684121549,
+                            0.000590995536185801,
+                            0.0002961631980724633,
+                            0.0004594426427502185,
+                            0.000973458809312433,
+                            0.0004766609345097095,
+                            0.0006601182976737618,
+                            0.001009256113320589,
+                            0.0006199177005328238,
+                            0.0010619658278301358,
+                            0.0009041576413437724,
+                            0.0006659405189566314,
+                            0.0009954854613170028,
+                            0.0005036531947553158,
+                            0.00031315715750679374,
+                            0.00038945337291806936,
+                            0.00031925912480801344,
+                            0.0003837370313704014,
+                            0.0003824240411631763,
+                            0.0003093581763096154,
+                            0.0003652931482065469,
+                            0.00028250087052583694,
+                            0.000446409045252949,
+                            0.00030261208303272724,
+                            0.0003413835074752569,
+                            0.00037125899689272046,
+                            0.0003780664410442114,
+                            0.0004145046405028552,
+                            0.0002826221170835197,
+                            0.0003647676785476506,
+                            0.00036313020973466337,
+                            0.00038279787986539304,
+                            0.00032129595638252795,
+                            0.0005206200876273215,
+                            0.00031237065559253097,
+                            0.0004489406128413975,
+                            0.000518923276104033,
+                            0.00034146688994951546,
+                            0.0008417475619353354,
+                            0.000455993867944926,
+                            0.00027934403624385595,
+                            0.00041721525485627353,
+                            0.0003220228827558458,
+                            0.000315069715725258,
+                            0.0003623533120844513,
+                            0.0002874818746931851,
+                            0.0002755798923317343,
+                            0.000493349158205092,
+                            0.0003560916520655155,
+                            0.0003290722088422626,
+                            0.00029343998176045716,
+                            0.00035123719135299325,
+                            0.00028371240478008986,
+                            0.0005136270774528384,
+                            0.00041564260027371347,
+                            0.0003135790175292641,
+                            0.0004223835130687803,
+                            0.00037552579306066036,
+                            0.00029142829589545727,
+                            0.00030514891841448843,
+                            0.00031148287234827876,
+                            0.0006540091126225889,
+                            0.0003340667753946036,
+                            0.0013886676169931889,
+                            0.0004242614086251706,
+                            0.00033520846045576036,
+                            0.00044702639570459723,
+                            0.0003447253839112818,
+                            0.00031183293322101235,
+                            0.00039092323277145624,
+                            0.0006050162483006716,
+                            0.00043617142364382744,
+                            0.0003953364794142544,
+                            0.000551801233086735,
+                            0.0004757955321110785,
+                            0.00042778244824148715,
+                            0.00040060223545879126,
+                            0.0003168497933074832,
+                            0.0003432183002587408,
+                            0.0006386396125890315,
+                            0.00028536474565044045,
+                            0.00035615809611044824,
+                            0.0005633338005281985,
+                            0.00033613471896387637,
+                            0.0010465403320267797,
+                            0.0009182884241454303,
+                            0.0009399338741786778,
+                            0.0003597665054257959,
+                            0.0002837585925590247,
+                            0.0006820152048021555,
+                            0.00030713490559719503,
+                            0.0013018284225836396,
+                            0.00034732307540252805,
+                            0.0002854691119864583,
+                            0.0003371539060026407,
+                            0.00034026714274659753,
+                            0.0007200830150395632,
+                            0.0013705536257475615,
+                            0.00036949149216525257,
+                            0.00043206571717746556,
+                            0.0006964067579247057,
+                            0.0010989574948325753,
+                            0.0004095171461813152,
+                            0.0018089727964252234,
+                            0.0006034019752405584,
+                            0.00035531033063307405,
+                            0.0003943347546737641,
+                            0.0004542810202110559,
+                            0.0003101035254076123,
+                            0.0003160785709042102,
+                            0.000986265717074275,
+                            0.0002834131009876728,
+                            0.00034173185122199357,
+                            0.0004907675320282578,
+                            0.0002829334989655763,
+                            0.000546852417755872,
+                            0.0005921353586018085,
+                            0.0002840326342266053,
+                            0.0004019230545964092,
+                            0.00032089397427625954,
+                            0.00043183646630495787,
+                            0.0003743019769899547,
+                            0.0003127091331407428,
+                            0.000402486533857882,
+                            0.0003629390266723931,
+                            0.00037208705907687545,
+                            0.0003081653267145157,
+                            0.00036477198591455817,
+                            0.0002814150939229876,
+                            0.00034692607005126774,
+                            0.0002926863671746105,
+                            0.00029097541118972003,
+                            0.0007037033210508525,
+                            0.00044232053915038705,
+                            0.0003572824352886528,
+                            0.000310061383061111,
+                            0.0008022090187296271,
+                            0.00030846905428916216,
+                            0.0004071817093063146,
+                            0.00032644777093082666,
+                            0.00028397279675118625,
+                            0.0005857293726876378,
+                            0.0003327687445562333,
+                            0.00028681752155534923,
+                            0.00034046557266265154,
+                            0.0002841993991751224,
+                            0.0008103878353722394,
+                            0.0005750342970713973,
+                            0.0004630962503142655,
+                            0.0003439740394242108,
+                            0.00030459443223662674,
+                            0.00030196644365787506,
+                            0.0005826058331876993,
+                            0.0003181980282533914,
+                            0.00031777238473296165,
+                            0.0003757030935958028,
+                            0.0006487558130174875,
+                            0.0004576435894705355,
+                            0.0004115883493795991,
+                            0.0009414332453161478,
+                            0.0007358000730164349,
+                            0.00035676162224262953,
+                            0.0003605106903705746,
+                            0.0006725829443894327,
+                            0.0002858435909729451,
+                            0.0010872766142711043,
+                            0.0002842164831236005,
+                            0.0003363105934113264,
+                            0.0005718077882193029,
+                            0.0003785539011005312,
+                            0.00043822548468597233,
+                            0.0003603281220421195,
+                            0.000587105518206954,
+                            0.00038203579606488347,
+                            0.00034344004234299064,
+                            0.00032804798684082925,
+                            0.0004533735627774149,
+                            0.0003978567838203162,
+                            0.0006468244246207178,
+                            0.0010332429083064198,
+                            0.0002912296331487596,
+                            0.0007861950434744358,
+                            0.00028411622042767704,
+                            0.00040638321661390364,
+                            0.0003162895154673606,
+                            0.0003273054026067257,
+                            0.00028509655385278165,
+                            0.00038218573899939656,
+                            0.00035553728230297565,
+                            0.0004800346796400845,
+                            0.0005484510911628604,
+                            0.000519435852766037,
+                            0.00036129774525761604,
+                            0.00046346569433808327,
+                            0.00032755814027041197,
+                            0.0013934740563854575,
+                            0.00037431626697070897,
+                            0.0013378127478063107,
+                            0.0005084291333332658,
+                            0.000847296672873199,
+                            0.0002841469249688089,
+                            0.0011704101925715804,
+                            0.0004140340315643698,
+                            0.00039367712452076375,
+                            0.0012057081330567598,
+                            0.00035917406785301864,
+                            0.0004335632547736168,
+                            0.00037223496474325657,
+                            0.0002950899361167103,
+                            0.000631263479590416,
+                            0.00031750815105624497,
+                            0.0012477696873247623,
+                            0.0003252011083532125,
+                            0.0008792679291218519,
+                            0.00029178275144658983,
+                            0.00028739715344272554,
+                            0.00042554817628115416,
+                            0.00041203724686056376,
+                            0.0003713291371241212,
+                            0.000284312991425395,
+                            0.000302668078802526,
+                            0.00039448586176149547,
+                            0.0012164171785116196,
+                            0.0014412520686164498,
+                            0.0004400519246701151,
+                            0.0004943209933117032,
+                            0.00038859338383190334,
+                            0.0014207124477252364,
+                            0.00031032843980938196,
+                            0.0005878882948309183,
+                            0.0005016736686229706,
+                            0.0014236283022910357,
+                            0.000324465538142249,
+                            0.000541280023753643,
+                            0.00036624071071855724,
+                            0.00030087330378592014,
+                            0.0003776578523684293,
+                            0.0003886290651280433,
+                            0.00034492145641706884,
+                            0.0002956623793579638,
+                            0.0005157773266546428,
+                            0.0005615065456368029,
+                            0.00031572047737427056,
+                            0.0004138678195886314,
+                            0.0003594051522668451,
+                            0.0003662500239443034,
+                            0.0003621582582127303,
+                            0.00032120762625709176,
+                            0.0014465638669207692,
+                            0.00029870372964069247,
+                            0.0006476548151113093,
+                            0.00028413618565537035,
+                            0.0006282738759182394,
+                            0.0003968009550590068,
+                            0.0005756476894021034,
+                            0.00029816917958669364,
+                            0.00036995657137595117,
+                            0.00033663230715319514,
+                            0.0005557225085794926,
+                            0.0003148395335301757,
+                            0.0004166025319136679,
+                            0.0006795573281124234,
+                            0.0009030423243530095,
+                            0.0004329317598603666,
+                            0.0005982198636047542,
+                            0.00036772340536117554,
+                            0.00031072896672412753,
+                            0.00028893930721096694,
+                            0.0006229123682714999,
+                            0.000676098105031997,
+                            0.0004370376409497112,
+                            0.00045093405060470104,
+                            0.0015917597338557243,
+                            0.0002840757661033422,
+                            0.0003998545871581882,
+                            0.0007623564451932907,
+                            0.0013914667069911957,
+                            0.0003652868326753378,
+                            0.00033102656016126275,
+                            0.0004921925719827414,
+                            0.0002800441288854927,
+                            0.00037786131724715233,
+                            0.000775708700530231,
+                            0.0004950299626216292,
+                            0.00029771728441119194,
+                            0.0005539105040952563,
+                            0.00046637747436761856,
+                            0.0006934449775144458,
+                            0.0003016884729731828,
+                            0.0003110957331955433,
+                            0.00037054219865240157,
+                            0.00028957551694475114,
+                            0.0004421452758833766,
+                            0.0007709823548793793,
+                            0.00044330660603009164,
+                            0.00028415993438102305,
+                            0.0003045845660381019,
+                            0.0002915471268352121,
+                            0.00035469350405037403,
+                            0.00036656149313785136,
+                            0.0003854346286971122,
+                            0.00032589383772574365,
+                            0.0004810814862139523,
+                            0.0003052228130400181,
+                            0.0003341003612149507,
+                            0.0005153461825102568,
+                            0.00048621889436617494,
+                            0.0009255969780497253,
+                            0.0002831385936588049,
+                            0.00028279039543122053,
+                            0.0003148738178424537,
+                            0.0002832153986673802,
+                            0.00031624833354726434,
+                            0.0005500881816260517,
+                            0.00036185976932756603,
+                            0.0011501549743115902,
+                            0.0003040115116164088,
+                            0.0007668976904824376,
+                            0.0006393985822796822,
+                            0.00032814196310937405,
+                            0.00033570441883057356,
+                            0.00031331542413681746,
+                            0.00037494010757654905,
+                            0.00036750180879607797,
+                            0.00034875288838520646,
+                            0.0002822776441462338,
+                            0.00034572844742797315,
+                            0.00044928063289262354,
+                            0.000710529217030853,
+                            0.0003993804566562176,
+                            0.00039756394107826054,
+                            0.0003009335487149656,
+                            0.00041941244853660464,
+                            0.00030926946783438325,
+                            0.0003127709496766329,
+                            0.00045215815771371126,
+                            0.0016698726685717702,
+                            0.0002839987282641232,
+                            0.0006586223025806248,
+                            0.00031769764609634876,
+                            0.0010002936469390988,
+                            0.0003235831973142922,
+                            0.0002842673275154084,
+                            0.00029392368742264807,
+                            0.0004512210434768349,
+                            0.0011668851366266608,
+                            0.0003059282316826284,
+                            0.00034811528166756034,
+                            0.00037107907701283693,
+                            0.0005203631590120494,
+                            0.0003267188440077007,
+                            0.0003623460070230067,
+                            0.0005439007072709501,
+                            0.0008478516829200089,
+                            0.000306726957205683,
+                            0.0003228371206205338,
+                            0.0005784339737147093,
+                            0.0004197088710498065,
+                            0.0003281014214735478,
+                            0.0003990264667663723,
+                            0.0004874153819400817,
+                            0.00028294764342717826,
+                            0.0004854108556173742,
+                            0.0003264122933614999,
+                            0.00031802544253878295,
+                            0.00035893634776584804,
+                            0.00045571234659291804,
+                            0.00041554332710802555,
+                            0.0005119455163367093,
+                            0.000377702439436689,
+                            0.0004589777672663331,
+                            0.0003128683310933411,
+                            0.000818530737888068,
+                            0.0005980133428238332,
+                            0.0002978772681672126,
+                            0.0006214017048478127,
+                            0.0013489959528669715,
+                            0.0010273581137880683,
+                            0.00036050891503691673,
+                            0.00057577172992751,
+                            0.0005534582305699587,
+                            0.00037081571645103395,
+                            0.0003415539104025811,
+                            0.00038381502963602543,
+                            0.001035062363371253,
+                            0.00031215607305057347,
+                            0.0002838981163222343,
+                            0.0003163365472573787,
+                            0.0003556004667188972,
+                            0.00030086151673458517,
+                            0.00034921171027235687,
+                            0.0005017706425860524,
+                            0.0008539429982192814,
+                            0.00032019062200561166,
+                            0.0004147432919126004,
+                            0.0009811004856601357,
+                            0.0002820272056851536,
+                            0.0008154803654178977,
+                            0.0014008620055392385,
+                            0.0006835553795099258,
+                            0.00028419055161066353,
+                            0.00048249680548906326,
+                            0.00033925333991646767,
+                            0.0010044919326901436,
+                            0.0002940058766398579,
+                            0.00029460538644343615,
+                            0.0006781094125472009,
+                            0.0003173912700731307,
+                            0.000273530458798632,
+                            0.0005599844735115767,
+                            0.00095893326215446,
+                            0.0002825171104632318,
+                            0.000339086982421577,
+                            0.0009670856525190175,
+                            0.00029748340602964163,
+                            0.0003358350077178329,
+                            0.0002957667747978121,
+                            0.00038788694655522704,
+                            0.00028400038718245924,
+                            0.000327861140249297,
+                            0.0011118302354589105,
+                            0.0002866291324608028,
+                            0.00038390056579373777,
+                            0.000294350553303957,
+                            0.0010991960298269987,
+                            0.0003322375996503979,
+                            0.0005055039073340595,
+                            0.00032294768607243896,
+                            0.00039533275412395597,
+                            0.00034672682522796094,
+                            0.0002842236135620624,
+                            0.0010673438664525747,
+                            0.0015169600956141949,
+                            0.0003100170870311558,
+                            0.00038805956137366593,
+                            0.00045883707934990525,
+                            0.0009962035110220313,
+                            0.0011825321707874537,
+                            0.0012418978149071336,
+                            0.00030115217668935657,
+                            0.0005039145471528172,
+                            0.0003718812658917159,
+                            0.0007222741842269897,
+                            0.00033552947570569813,
+                            0.0005116912070661783,
+                            0.00044199751573614776,
+                            0.000334005628246814,
+                            0.00031480489997193217,
+                            0.0003973045386373997,
+                            0.0005989025812596083,
+                            0.0004655934462789446,
+                            0.0004218815010972321,
+                            0.0005356041947379708,
+                            0.000799138389993459,
+                            0.00037928030360490084,
+                            0.0013436180306598544,
+                            0.0002892886695917696,
+                            0.00039461994310840964,
+                            0.00029496909701265395,
+                            0.00031105964444577694,
+                            0.0004741048614960164,
+                            0.0003768939641304314,
+                            0.0002821700763888657,
+                            0.0005084298318251967,
+                            0.0005518884863704443,
+                            0.0009598908363841474,
+                            0.0005035848589614034,
+                            0.00030653149588033557,
+                            0.0004287103656679392,
+                            0.0004224555741529912,
+                            0.001149530173279345,
+                            0.000310674513457343,
+                            0.0003424111055210233,
+                            0.0003483722102828324,
+                            0.0003946165379602462,
+                            0.0011310321278870106,
+                            0.00039849907625466585,
+                            0.0005550671485252678,
+                            0.00034450911334715784,
+                            0.000332857045577839,
+                            0.0004087038105353713,
+                            0.00028350090724416077,
+                            0.00033054326195269823,
+                            0.00038012219010852277,
+                            0.00034405841142870486,
+                            0.0008729264372959733,
+                            0.0002719834737945348,
+                            0.000987522886134684,
+                            0.0003084120689891279,
+                            0.00028739229310303926,
+                            0.0004512454033829272,
+                            0.00042237850720994174,
+                            0.0003716879291459918,
+                            0.000409138563554734,
+                            0.00043003205792047083,
+                            0.0005528809851966798,
+                            0.0003497044672258198,
+                            0.00028379997820593417,
+                            0.0003860352444462478,
+                            0.00045692361891269684,
+                            0.0003742114349734038,
+                            0.0004412681155372411,
+                            0.0005044919671490788,
+                            0.00032041114172898233,
+                            0.00044864954543299973,
+                            0.0003904080658685416,
+                            0.00028289761394262314,
+                            0.0004997641080990434,
+                            0.0005708723911084235,
+                            0.0003953521081712097,
+                            0.0002814438194036484,
+                            0.0003303053672425449,
+                            0.00048263309872709215,
+                            0.00041475173202343285,
+                            0.000402385980123654,
+                            0.0007938311318866909,
+                            0.00033998556318692863,
+                            0.00029798460309393704,
+                            0.0003286547143943608,
+                            0.0007494182209484279,
+                            0.0007150197052396834,
+                            0.00034966770908795297,
+                            0.00028969175764359534,
+                            0.0002962248690892011,
+                            0.0004567696887534112,
+                            0.0012691004667431116,
+                            0.0008008708828128874,
+                            0.0003948856028728187,
+                            0.0004084133543074131,
+                            0.0011987333418801427,
+                            0.0005854872870258987,
+                            0.0005985163734294474,
+                            0.000366822729120031,
+                            0.0003883333411067724,
+                            0.0003778077370952815,
+                            0.0003651421284303069,
+                            0.0008245871867984533,
+                            0.00043467950308695436,
+                            0.00037905742647126317,
+                            0.0004623558488674462,
+                            0.00030537089332938194,
+                            0.0012707198038697243,
+                            0.00035383415524847806,
+                            0.0005610642256215215,
+                            0.00043364689918234944,
+                            0.0008084459695965052,
+                            0.00030862490530125797,
+                            0.0003432256926316768,
+                            0.00029391530551947653,
+                            0.0003136513114441186,
+                            0.00032232931698672473,
+                            0.00034371548099443316,
+                            0.0003910584491677582,
+                            0.000448949373094365,
+                            0.0003270484448876232,
+                            0.0012054800754413009,
+                            0.00039704167284071445,
+                            0.0002826870768330991,
+                            0.00027986723580397666,
+                            0.0005242979386821389,
+                            0.0005718782776966691,
+                            0.0005337299662642181,
+                            0.00035788395325653255,
+                            0.00028398571885190904,
+                            0.0003010717628058046,
+                            0.0002805494877975434,
+                            0.0003779872495215386,
+                            0.0003770413459278643,
+                            0.0006498018046841025,
+                            0.0006757869268767536,
+                            0.00038514251355081797,
+                            0.0005091438652016222,
+                            0.00048765886458568275,
+                            0.0011039585806429386,
+                            0.00047394941793754697,
+                            0.0013679106486961246,
+                            0.0003456230915617198,
+                            0.0008132326765917242,
+                            0.0003187145630363375,
+                            0.0006579459295608103,
+                            0.0003251216548960656,
+                            0.0002817421336658299,
+                            0.0004063551314175129,
+                            0.0003240901860408485,
+                            0.00039437811938114464,
+                            0.0004517536435741931,
+                            0.00032125008874572814,
+                            0.0003164161171298474,
+                            0.00028868971276097,
+                            0.00032044469844549894,
+                            0.0003500424209050834,
+                            0.0009798426181077957,
+                            0.00039046333404257894,
+                            0.0003754232020583004,
+                            0.00028221242246218026,
+                            0.00028492705314420164,
+                            0.0005728930118493736,
+                            0.0003539453027769923,
+                            0.00034472395782358944,
+                            0.0002939954283647239,
+                            0.0004031058051623404,
+                            0.000390353990951553,
+                            0.0009987520752474666,
+                            0.00027879743720404804,
+                            0.0003225197142455727,
+                            0.0002902976702898741,
+                            0.0008709267131052911,
+                            0.00035792423295788467,
+                            0.00039898965042084455,
+                            0.0008099389378912747,
+                            0.0003260693629272282,
+                            0.0002840018132701516,
+                            0.0004072043811902404,
+                            0.0003544651553966105,
+                            0.0003588576801121235,
+                            0.0009605203522369266,
+                            0.0006362112471833825,
+                            0.00031490877154283226,
+                            0.00030914737726561725,
+                            0.00031555292662233114,
+                            0.0002883070264942944,
+                            0.0003112491685897112,
+                            0.0008677987498231232,
+                            0.00028391153318807483,
+                            0.0005494318902492523,
+                            0.000315994955599308,
+                            0.000502966286148876,
+                            0.00029876004555262625,
+                            0.0003736657090485096,
+                            0.0003424585738684982,
+                            0.00034811694058589637,
+                            0.00034087401581928134,
+                            0.0007476913160644472,
+                            0.0004458984767552465,
+                            0.0006608339608646929,
+                            0.00028246408328413963,
+                            0.0003746967122424394,
+                            0.0004028499242849648,
+                            0.0002830054145306349,
+                            0.0002835380146279931,
+                            0.00041178910760208964,
+                            0.0009342003613710403,
+                            0.0002840802480932325,
+                            0.0003264328988734633,
+                            0.0003914860135409981,
+                            0.00030066759791225195,
+                            0.000660661025904119,
+                            0.000284134061075747,
+                            0.0003243459213990718,
+                            0.000396559335058555,
+                            0.0008159998687915504,
+                            0.000606907473411411,
+                            0.0005772107979282737,
+                            0.0008545679738745093,
+                            0.0002858621592167765,
+                            0.0002838163054548204,
+                            0.0003475831763353199,
+                            0.0006000676075927913,
+                            0.00030404538847506046,
+                            0.0003786980814766139,
+                            0.0003013213281519711,
+                            0.0004613413766492158,
+                            0.0003966796211898327,
+                            0.0011931636836379766,
+                            0.0004714869719464332,
+                            0.00037482570041902363,
+                            0.0004359978483989835,
+                            0.00033747623092494905,
+                            0.00028338085394352674,
+                            0.000773243373259902,
+                            0.00044527996215038,
+                            0.0002767015539575368,
+                            0.00034279649844393134,
+                            0.0005829644505865872,
+                            0.0005940302507951856,
+                            0.0003572629939299077,
+                            0.0005558438715524971,
+                            0.0007201439584605396,
+                            0.001107542309910059,
+                            0.00045914106885902584,
+                            0.00044584242277778685,
+                            0.0002838438958860934,
+                            0.0003566427913028747,
+                            0.0003825913881883025,
+                            0.0010580099187791348,
+                            0.0002832348400261253,
+                            0.0004967327695339918,
+                            0.00032422278309240937,
+                            0.0003692337777465582,
+                            0.00028385326731950045,
+                            0.0002833828912116587,
+                            0.00033788979635573924,
+                            0.0003076098801102489,
+                            0.000954833987634629,
+                            0.0005589376087300479,
+                            0.000284095061942935,
+                            0.0003763312997762114,
+                            0.0003971692640334368,
+                            0.0005298887263052166,
+                            0.00038265035254880786,
+                            0.0008445592829957604,
+                            0.0005935361259616911,
+                            0.0003372906649019569,
+                            0.00045906155719421804,
+                            0.001314978115260601,
+                            0.00038449798012152314,
+                            0.000419290125137195,
+                            0.0003291791945230216,
+                            0.0006952360854484141,
+                            0.0002825061965268105,
+                            0.0007897500763647258,
+                            0.0003534822608344257,
+                            0.00037543821963481605,
+                            0.0002949435729533434,
+                            0.0005223778425715864,
+                            0.0002830005541909486,
+                            0.00031025498174130917,
+                            0.0003285235143266618,
+                            0.0005214841803535819,
+                            0.000288611016003415,
+                            0.00030876704840920866,
+                            0.0002839976514223963,
+                            0.0005216882564127445,
+                            0.0005970035563223064,
+                            0.0006343641434796154,
+                            0.00038059617509134114,
+                            0.0005400538211688399,
+                            0.00028418860165402293,
+                            0.0002926556917373091,
+                            0.00030343420803546906,
+                            0.0004006662347819656,
+                            0.00028366412152536213,
+                            0.0003794259682763368,
+                            0.0003900653391610831,
+                            0.0002857886429410428,
+                            0.0005211834795773029,
+                            0.0006416778778657317,
+                            0.00028405978810042143,
+                            0.0002960910205729306,
+                            0.001205215696245432,
+                            0.0009626834653317928,
+                            0.0009339397656731308,
+                            0.0005446718423627317,
+                            0.0004703763988800347,
+                            0.0005982076399959624,
+                            0.00034224241971969604,
+                            0.00028484570793807507,
+                            0.0008267127559520304,
+                            0.00031526139355264604,
+                            0.00039423289126716554,
+                            0.00043255052878521383,
+                            0.00035081940586678684,
+                            0.0006016147672198713,
+                            0.00033123482717201114,
+                            0.000762397947255522,
+                            0.0003929774393327534,
+                            0.0003153483266942203,
+                            0.0002933311916422099,
+                            0.0009992996929213405,
+                            0.0005841218517161906,
+                            0.0002887049922719598,
+                            0.0003462299646344036,
+                            0.0014223942998796701,
+                            0.0008322074427269399,
+                            0.0003644975950010121,
+                            0.0002835464256349951,
+                            0.0002833905164152384,
+                            0.0003304620331618935,
+                            0.00032803736394271255,
+                            0.00040473503759130836,
+                            0.001126119284890592,
+                            0.00028950252453796566,
+                            0.0013206035364419222,
+                            0.0003087140212301165,
+                            0.0004055930767208338,
+                            0.0007163461414165795,
+                            0.00034983688965439796,
+                            0.0004292833327781409,
+                            0.0003190033894497901,
+                            0.0006110852118581533,
+                            0.0003077608998864889,
+                            0.00040533204446546733,
+                            0.0005292289424687624
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 74,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.06937800347805023
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "over_arch.model.0._mlp.1",
+                "input_tensor_infos": [
+                    {
+                        "id": 74,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.06937800347805023
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 75,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.056720979511737823
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 75,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.056720979511737823
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "over_arch:model:0:_mlp:2": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "over_arch.model.0._mlp.2._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 75,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.056720979511737823
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0008439983939751983,
+                            0.0008375866455025971,
+                            0.0008955321391113102,
+                            0.0008798613562248647,
+                            0.0006287649739533663,
+                            0.00028428243240341544,
+                            0.0004326157795730978,
+                            0.0008682458428665996,
+                            0.0008171977824531496,
+                            0.0007652925560250878,
+                            0.0007299039280042052,
+                            0.00028422329341992736,
+                            0.0002837271604221314,
+                            0.0008257727604359388,
+                            0.000546636525541544,
+                            0.00028413746622391045,
+                            0.0011459465604275465,
+                            0.0008761470671743155,
+                            0.0009948251536116004,
+                            0.0002840450615622103,
+                            0.00033815871574915946,
+                            0.00028380905860103667,
+                            0.0007236592937260866,
+                            0.0006519087473861873,
+                            0.0005134690436534584,
+                            0.0007675826782360673,
+                            0.0008623443427495658,
+                            0.0008589957724325359,
+                            0.000354636344127357,
+                            0.0009514286066405475,
+                            0.0007339451694861054,
+                            0.0007231978815980256,
+                            0.0004992815083824098,
+                            0.00028392343665473163,
+                            0.0009282110258936882,
+                            0.00121181586291641,
+                            0.0007466144161298871,
+                            0.00031051423866301775,
+                            0.0008927964372560382,
+                            0.0010426839580759406,
+                            0.00025596944033168256,
+                            0.0008145669125951827,
+                            0.0004765187914017588,
+                            0.0003277422802057117,
+                            0.000466895813588053,
+                            0.0011705546639859676,
+                            0.0002841096429619938,
+                            0.00032896213815547526,
+                            0.000691102584823966,
+                            0.0003178451443091035,
+                            0.0009320061071775854,
+                            0.0003315807261969894,
+                            0.000576640188228339,
+                            0.0006685220869258046,
+                            0.0003173323639202863,
+                            0.0009944038465619087,
+                            0.0008982695871964097,
+                            0.0005172552773728967,
+                            0.0005110985366627574,
+                            0.0005825958214700222,
+                            0.0002840439847204834,
+                            0.0004882380017079413,
+                            0.0003617404727265239,
+                            0.001307002268731594,
+                            0.00028215511702001095,
+                            0.0005768366972915828,
+                            0.00028283149003982544,
+                            0.000313000928144902,
+                            0.00028281789855100214,
+                            0.0007240991690196097,
+                            0.0013267698232084513,
+                            0.0007963213138282299,
+                            0.0002818229258991778,
+                            0.00028354188543744385,
+                            0.00031837105052545667,
+                            0.0006534203421324492,
+                            0.0007820220198482275,
+                            0.0007862296770326793,
+                            0.0008049567695707083,
+                            0.0009008807828649879,
+                            0.00040039687883108854,
+                            0.0006293424521572888,
+                            0.00046719086822122335,
+                            0.00106506806332618,
+                            0.0011204335605725646,
+                            0.0012118471786379814,
+                            0.0004871907876804471,
+                            0.0007480383501388133,
+                            0.0002810987934935838,
+                            0.0007383385091088712,
+                            0.0005137349362485111,
+                            0.00037240266101434827,
+                            0.0005064517026767135,
+                            0.0006127819651737809,
+                            0.0005657864385284483,
+                            0.00029815343441441655,
+                            0.0009226286201737821,
+                            0.0003265518753323704,
+                            0.00040829047793522477,
+                            0.0008929641917347908,
+                            0.0008704004576429725,
+                            0.00030161510221660137,
+                            0.0005714913131669164,
+                            0.0003119362227153033,
+                            0.0002831604506354779,
+                            0.000917614612262696,
+                            0.000383055885322392,
+                            0.0008262323099188507,
+                            0.00035608361940830946,
+                            0.0003505531349219382,
+                            0.00048199237789958715,
+                            0.0005073208594694734,
+                            0.00042606479837559164,
+                            0.0004443845245987177,
+                            0.0007431705016642809,
+                            0.00067929393844679,
+                            0.0007416713633574545,
+                            0.0007070955471135676,
+                            0.0002993532398249954,
+                            0.0003164989175274968,
+                            0.0012267710408195853,
+                            0.00028391965315677226,
+                            0.00040847109630703926,
+                            0.00031179338111542165,
+                            0.0009427855839021504,
+                            0.000545342278201133,
+                            0.0010769644286483526,
+                            0.0007624970166943967,
+                            0.0009466302581131458,
+                            0.0004609548195730895,
+                            0.0007269855123013258,
+                            0.0009209975833073258,
+                            0.0013785982737317681,
+                            0.0008056023507378995,
+                            0.0004976672353222966,
+                            0.0009077205322682858,
+                            0.000757250003516674,
+                            0.000391686538932845,
+                            0.0006011815275996923,
+                            0.000582993496209383,
+                            0.0008118650875985622,
+                            0.0008245312492363155,
+                            0.0006808532634750009,
+                            0.0007451513083651662,
+                            0.00028412393294274807,
+                            0.0007617655210196972,
+                            0.0007903924561105669,
+                            0.0006569374818354845,
+                            0.0007280218414962292,
+                            0.0008341351058334112,
+                            0.0006980453035794199,
+                            0.0003343485004734248,
+                            0.0005598588613793254,
+                            0.0003152242861688137,
+                            0.00041998494998551905,
+                            0.0007840343751013279,
+                            0.000741552677936852,
+                            0.001214325544424355,
+                            0.000774384243413806,
+                            0.0006996843148954213,
+                            0.000926548964343965,
+                            0.00044981404789723456,
+                            0.0010098058264702559,
+                            0.0002931940252892673,
+                            0.0002838077489286661,
+                            0.000752918713260442,
+                            0.0008225596975535154,
+                            0.000832980964332819,
+                            0.0011421074159443378,
+                            0.0008823562529869378,
+                            0.0008769393316470087,
+                            0.0005185242043808103,
+                            0.00038256627158261836,
+                            0.0006194222951307893,
+                            0.00071516796015203,
+                            0.0010526147671043873,
+                            0.0008414768963120878,
+                            0.0005736786988563836,
+                            0.0005174560355953872,
+                            0.0008611251250840724,
+                            0.0009171862620860338,
+                            0.000848720723297447,
+                            0.0008595816907472908,
+                            0.00042038242099806666,
+                            0.00028183599351905286,
+                            0.000369997724192217,
+                            0.0002839472726918757,
+                            0.0008393801981583238,
+                            0.0005790501600131392,
+                            0.0005708873504772782,
+                            0.0008983287261798978,
+                            0.0008916883380152285,
+                            0.0002802104572765529,
+                            0.0002924542932305485,
+                            0.00028413356631062925,
+                            0.0008127503097057343,
+                            0.0004449718398973346,
+                            0.0012414513621479273,
+                            0.00028340143035165966,
+                            0.0005271690897643566,
+                            0.0005483786226250231,
+                            0.00032253313111141324,
+                            0.0003174988378304988,
+                            0.0009066020720638335,
+                            0.00031091104028746486,
+                            0.00040098626050166786,
+                            0.0009794048964977264,
+                            0.0004730988875962794,
+                            0.0008551410865038633,
+                            0.0008251644321717322,
+                            0.0002810214937198907,
+                            0.001175140612758696,
+                            0.0008976194658316672,
+                            0.0013498462503775954,
+                            0.0005527359317056835,
+                            0.0004243158327881247,
+                            0.0002806395059451461,
+                            0.0003081021422985941,
+                            0.00036082518636249006,
+                            0.0008515979861840606,
+                            0.0005800520302727818,
+                            0.0003668942954391241,
+                            0.0006526021752506495,
+                            0.0003887979837600142,
+                            0.0006980531034059823,
+                            0.0002832484897226095,
+                            0.0008472532499581575,
+                            0.0003043880278710276,
+                            0.0008816769695840776,
+                            0.0007668957696296275,
+                            0.0002958235563710332,
+                            0.00039483970613218844,
+                            0.0008113874355331063,
+                            0.0002792721497826278,
+                            0.0008082781569100916,
+                            0.0008452177280560136,
+                            0.0007958398200571537,
+                            0.000472505169454962,
+                            0.00028405041666701436,
+                            0.0006203328957781196,
+                            0.0003036067937500775,
+                            0.0002841488749254495,
+                            0.0008782275835983455,
+                            0.00040855410043150187,
+                            0.000620305712800473,
+                            0.00031694644712843,
+                            0.0005220544408075511,
+                            0.0007439623004756868,
+                            0.0005033946363255382,
+                            0.00030082176090218127,
+                            0.0007605973514728248,
+                            0.0004062239604536444,
+                            0.0002817029890138656,
+                            0.0002939444675575942,
+                            0.0005928307655267417,
+                            0.0011546647874638438,
+                            0.0004981259698979557,
+                            0.0005122735747136176,
+                            0.00028302305145189166,
+                            0.0012001384748145938,
+                            0.0007256009848788381,
+                            0.0003244933614041656,
+                            0.0009523342596367002,
+                            0.0002820776426233351,
+                            0.001062694238498807,
+                            0.0011888252338394523,
+                            0.0005883833509869874,
+                            0.0011953189969062805,
+                            0.0009743029368110001,
+                            0.000802914728410542,
+                            0.0006656505283899605,
+                            0.0008144258754327893,
+                            0.0013241334818303585,
+                            0.000737140653654933,
+                            0.00028384249890223145,
+                            0.00039510370697826147,
+                            0.00028379293507896364,
+                            0.0004821329202968627,
+                            0.001067762030288577,
+                            0.00029776431620121,
+                            0.0008812648011371493,
+                            0.000991176930256188,
+                            0.0003090747050009668,
+                            0.000983735080808401,
+                            0.0008521975832991302,
+                            0.0006820394191890955,
+                            0.0010724645107984543,
+                            0.0011649421649053693,
+                            0.00031403411412611604,
+                            0.0003988122916780412,
+                            0.0005805958644486964,
+                            0.00029082148103043437,
+                            0.00028380510048009455,
+                            0.0006972128758206964,
+                            0.0003155519952997565,
+                            0.0004785101627930999,
+                            0.0010767501080408692,
+                            0.0007906444370746613,
+                            0.0006622430519200861,
+                            0.0007529315771535039,
+                            0.00026544739375822246,
+                            0.0004225709126330912,
+                            0.0003070734383072704,
+                            0.00094658846501261,
+                            0.0005850552697665989,
+                            0.0004017903993371874,
+                            0.0009481224697083235,
+                            0.0002782227529678494,
+                            0.0008687381050549448,
+                            0.0008059766259975731,
+                            0.0007994582992978394,
+                            0.0002881997497752309,
+                            0.0006663536187261343,
+                            0.000888931390363723,
+                            0.00031791007495485246,
+                            0.0008930846233852208,
+                            0.000848433468490839,
+                            0.0006166084203869104,
+                            0.0008067970047704875,
+                            0.0007067914120852947,
+                            0.0005035471986047924,
+                            0.0009086164063774049,
+                            0.00028376677073538303,
+                            0.0002837764041032642,
+                            0.0008905113791115582,
+                            0.001312612323090434,
+                            0.0010286009637638927,
+                            0.0007562097162008286,
+                            0.0003100709873251617,
+                            0.0009933600667864084,
+                            0.0002826208947226405,
+                            0.00028303274302743375,
+                            0.0008196637500077486,
+                            0.0008654700359329581,
+                            0.0003056610294152051,
+                            0.0007050114800222218,
+                            0.0010540548246353865,
+                            0.0008662730688229203,
+                            0.0002919977705460042,
+                            0.000673942151479423,
+                            0.0002798472414724529,
+                            0.0007562153623439372,
+                            0.0002836628118529916,
+                            0.000568636751268059,
+                            0.0006539009627886117,
+                            0.0007224729051813483,
+                            0.0002841288805939257,
+                            0.0005804966785944998,
+                            0.0003236926277168095,
+                            0.0002842941612470895,
+                            0.0004317769780755043,
+                            0.0008142548031173646,
+                            0.00028428269433788955,
+                            0.0005945849698036909,
+                            0.0009419991984032094,
+                            0.0003246825363021344,
+                            0.0008956872625276446,
+                            0.0004573516780510545,
+                            0.0008774998714216053,
+                            0.0002826752024702728,
+                            0.0008036344079300761,
+                            0.0007771042874082923,
+                            0.00039299007039517164,
+                            0.0006828919285908341,
+                            0.0002824591938406229,
+                            0.0005631931126117706,
+                            0.0004196437366772443,
+                            0.00031080571352504194,
+                            0.0006402449798770249,
+                            0.0006263347459025681,
+                            0.0007142101530916989,
+                            0.00028319485136307776,
+                            0.0013254168443381786,
+                            0.0003126136725768447,
+                            0.0005467491573654115,
+                            0.0006208597915247083,
+                            0.00043606245890259743,
+                            0.0006219617207534611,
+                            0.0010824446799233556,
+                            0.0009390188497491181,
+                            0.0006225431570783257,
+                            0.0002782248775474727,
+                            0.0007791525567881763,
+                            0.0008890003082342446,
+                            0.0010234923101961613,
+                            0.0006928790244273841,
+                            0.00044114718912169337,
+                            0.0005289486143738031,
+                            0.0011683127377182245,
+                            0.00028280357946641743,
+                            0.0005783013766631484,
+                            0.0007025353843346238,
+                            0.0009078230359591544,
+                            0.00029316666768863797,
+                            0.0008242691401392221,
+                            0.0008286824449896812,
+                            0.00093989132437855,
+                            0.00028363714227452874,
+                            0.0007666872697882354,
+                            0.0005125685129314661,
+                            0.00027747778221964836,
+                            0.0002841285604517907,
+                            0.0008731934940442443,
+                            0.0009969169041141868,
+                            0.0003125716757494956,
+                            0.0007498744525946677,
+                            0.0009580985642969608,
+                            0.0009631147258915007,
+                            0.0008296136511489749,
+                            0.00028006156207993627,
+                            0.0007983554969541728,
+                            0.0002837991341948509,
+                            0.00035484350519254804,
+                            0.0008363404776901007,
+                            0.0007787960930727422,
+                            0.0008676800061948597,
+                            0.0004944662796333432,
+                            0.0005173769313842058,
+                            0.0007506338879466057,
+                            0.0008222070173360407,
+                            0.00030901553691364825,
+                            0.0009088328806683421,
+                            0.0011975232046097517,
+                            0.000684687402099371,
+                            0.0009031188674271107,
+                            0.000739680603146553,
+                            0.0007988765719346702,
+                            0.0005565728060901165,
+                            0.0004230242338962853,
+                            0.0010988230351358652,
+                            0.0011044363491237164,
+                            0.0005189694929867983,
+                            0.00046950491378083825,
+                            0.0008452783222310245,
+                            0.0010664877481758595,
+                            0.0006470533553510904,
+                            0.000743334589060396,
+                            0.0002841344103217125,
+                            0.0007255879463627934,
+                            0.0002840875240508467,
+                            0.00030698179034516215,
+                            0.0008848202414810658,
+                            0.0003039991424884647,
+                            0.0003141624911222607,
+                            0.0007275085663422942,
+                            0.0005124829476699233,
+                            0.0003937497385777533,
+                            0.0011681077303364873,
+                            0.0008373431628569961,
+                            0.0002840848173946142,
+                            0.0010523161618039012,
+                            0.0008332165307365358,
+                            0.0005403229151852429,
+                            0.0003705461276695132,
+                            0.00033838392118923366,
+                            0.0002839824592228979,
+                            0.00026925417478196323,
+                            0.0009653708548285067,
+                            0.0003132237179670483,
+                            0.0007052713772282004,
+                            0.00028352910885587335,
+                            0.0008807480335235596,
+                            0.0008750634151510894,
+                            0.0003619352064561099,
+                            0.00028353248490020633,
+                            0.0003067194775212556,
+                            0.0006489321822300553,
+                            0.0003229225112590939,
+                            0.0002786071563605219,
+                            0.0007422027410939336,
+                            0.0002834499755408615,
+                            0.00047504896065220237,
+                            0.000283747969660908,
+                            0.00032070925226435065,
+                            0.0007455456652678549,
+                            0.0005492044147104025,
+                            0.00035631840000860393,
+                            0.00034908283851109445,
+                            0.000809166522230953,
+                            0.0005602791206911206,
+                            0.000283917790511623,
+                            0.0005345066310837865,
+                            0.00028285992448218167,
+                            0.000714900204911828,
+                            0.0006879022112116218,
+                            0.0007103677489794791,
+                            0.00029479467775672674,
+                            0.00038468686398118734,
+                            0.0002829564909916371,
+                            0.00039941942668519914,
+                            0.0009034191025421023,
+                            0.00035901187220588326,
+                            0.0007604579441249371,
+                            0.0004967549466527998,
+                            0.0006574101280421019,
+                            0.0008649209048599005,
+                            0.0009955903515219688,
+                            0.0004360152524895966,
+                            0.00036700430791825056,
+                            0.0009192851721309125,
+                            0.0003421066503506154,
+                            0.0007388992817141116,
+                            0.0008827022975310683,
+                            0.0009092362597584724,
+                            0.00031470664544031024,
+                            0.0008488845778629184,
+                            0.0007438204484060407,
+                            0.0002820078225340694,
+                            0.0003230929432902485,
+                            0.0004955327603965998,
+                            0.0007208894821815193,
+                            0.0008669632370583713
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 76,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.029124625027179718
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "over_arch.model.0._mlp.2",
+                "input_tensor_infos": [
+                    {
+                        "id": 76,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.029124625027179718
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 77,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.029578574001789093
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 77,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.029578574001789093
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    },
+    "over_arch:model:0:_mlp:3": {
+        "q_op_infos": {
+            "0": {
+                "op_type": "<class 'torch.nn.modules.linear.Linear'>",
+                "op_type_is_module": true,
+                "fqn": "over_arch.model.0._mlp.3._linear",
+                "input_tensor_infos": [
+                    {
+                        "id": 77,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "force_dtype": "torch.qint8",
+                        "scale": [
+                            0.029578574001789093
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [
+                    {
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.0009896227857097983,
+                            0.0008572051883675158,
+                            0.001198483514599502,
+                            0.0005474613863043487,
+                            0.0007853093557059765,
+                            0.0003459702420514077,
+                            0.0005285667139105499,
+                            0.0006019636057317257,
+                            0.0005676979781128466,
+                            0.00088829779997468,
+                            0.00042100451537407935,
+                            0.0008653337717987597,
+                            0.0003851456567645073,
+                            0.0003856007242575288,
+                            0.0006366443121805787,
+                            0.00042315881000831723,
+                            0.00038480176590383053,
+                            0.0006913369870744646,
+                            0.0004370885726530105,
+                            0.000377921387553215,
+                            0.00042368561844341457,
+                            0.0003914364206138998,
+                            0.0003459407016634941,
+                            0.001088490360416472,
+                            0.00037055157008580863,
+                            0.0003844711754936725,
+                            0.0011346719693392515,
+                            0.00038346595829352736,
+                            0.0003817911201622337,
+                            0.001251233508810401,
+                            0.0009258828358724713,
+                            0.0009445968316867948,
+                            0.0007541850791312754,
+                            0.00034504258655942976,
+                            0.0011202692985534668,
+                            0.001131769153289497,
+                            0.0004935032920911908,
+                            0.00042065518209710717,
+                            0.0006229162681847811,
+                            0.0003941462200600654,
+                            0.00042143696919083595,
+                            0.00038572450284846127,
+                            0.0003830971254501492,
+                            0.0005445811548270285,
+                            0.0007021193159744143,
+                            0.0005052977940067649,
+                            0.00038530901656486094,
+                            0.0004467799444682896,
+                            0.0003840288263745606,
+                            0.00042382569517940283,
+                            0.0007886026287451386,
+                            0.0004892142023891211,
+                            0.000380529003450647,
+                            0.0003838090051431209,
+                            0.0003840307763312012,
+                            0.0008180958684533834,
+                            0.0005084608565084636,
+                            0.0010938532650470734,
+                            0.0009341147961094975,
+                            0.0003807917528320104,
+                            0.000346017477568239,
+                            0.000607560679782182,
+                            0.00040796419489197433,
+                            0.00043737958185374737,
+                            0.0007032243302091956,
+                            0.0005364107200875878,
+                            0.00041802445775829256,
+                            0.0010367007926106453,
+                            0.0005452494951896369,
+                            0.0005624445620924234,
+                            0.00045300161582417786,
+                            0.001162357977591455,
+                            0.0003997426538262516,
+                            0.0006047710194252431,
+                            0.0003785329172387719,
+                            0.0003850387001875788,
+                            0.0007558690849691629,
+                            0.0005747118848375976,
+                            0.0005248531815595925,
+                            0.0010629676980897784,
+                            0.0009757582447491586,
+                            0.0006186033133417368,
+                            0.0006422698497772217,
+                            0.0003857734554912895,
+                            0.0010658795945346355,
+                            0.0004402318736538291,
+                            0.0006710501038469374,
+                            0.0006768640596419573,
+                            0.0005102168070152402,
+                            0.0010618554661050439,
+                            0.0003836831310763955,
+                            0.000380586861865595,
+                            0.0004883729270659387,
+                            0.0006554708816111088,
+                            0.00034602644154801965,
+                            0.00112715398427099,
+                            0.0010258523980155587,
+                            0.0007345776539295912,
+                            0.0007758919382467866,
+                            0.0004017517785541713,
+                            0.0007698514964431524,
+                            0.0003834134549833834,
+                            0.0009562054183334112,
+                            0.0003815741220023483,
+                            0.0003838565025944263,
+                            0.0007083253585733473,
+                            0.0009754913044162095,
+                            0.0003845460887532681,
+                            0.0008874566410668194,
+                            0.0006715938216075301,
+                            0.0008709249086678028,
+                            0.00038498506182804704,
+                            0.0007783515611663461,
+                            0.0007626043516211212,
+                            0.0006294058403000236,
+                            0.0003905841731466353,
+                            0.00047790506505407393,
+                            0.00038312660763040185,
+                            0.0003846437612082809,
+                            0.0006368663161993027,
+                            0.000979395816102624,
+                            0.0005891987239010632,
+                            0.0006237659836187959,
+                            0.0003866555343847722,
+                            0.0007980099180713296,
+                            0.00038450612919405103,
+                            0.001103765331208706,
+                            0.0004372021066956222,
+                            0.0005261925398372114,
+                            0.0010065215174108744,
+                            0.0009673194144852459,
+                            0.0008421838865615427,
+                            0.0005378860514611006,
+                            0.00038449515704996884,
+                            0.0004704183083958924,
+                            0.0012787490850314498,
+                            0.0009182753274217248,
+                            0.0004454058944247663,
+                            0.001161054358817637,
+                            0.000486403499962762,
+                            0.00038290489464998245,
+                            0.001173810800537467,
+                            0.0007968087447807193,
+                            0.0007100076181814075,
+                            0.000997997703962028,
+                            0.0012705534463748336,
+                            0.0003669980796985328,
+                            0.0004196249647065997,
+                            0.0003880627918988466,
+                            0.0007518680649809539,
+                            0.0011181592708453536,
+                            0.00047980283852666616,
+                            0.0009714037296362221,
+                            0.0009441187139600515,
+                            0.0011748512042686343,
+                            0.0003820068668574095,
+                            0.0004048717673867941,
+                            0.0008493363275192678,
+                            0.00048185218474827707,
+                            0.0005296703893691301,
+                            0.0010345730697736144,
+                            0.0004804489144589752,
+                            0.0008544373558834195,
+                            0.00038488549762405455,
+                            0.0007415388827212155,
+                            0.0007802376057952642,
+                            0.0011918494710698724,
+                            0.000383366656024009,
+                            0.0003795800730586052,
+                            0.0007171324105001986,
+                            0.0010234189685434103,
+                            0.00041104640695266426,
+                            0.0003985948278568685,
+                            0.0008414819021709263,
+                            0.001138294581323862,
+                            0.0004646815068554133,
+                            0.0012021927395835519,
+                            0.0005322525394149125,
+                            0.00044249510392546654,
+                            0.00042809962178580463,
+                            0.0009862397564575076,
+                            0.000647339504212141,
+                            0.0003850024950224906,
+                            0.00038538951775990427,
+                            0.0005029404419474304,
+                            0.0013585997512564063,
+                            0.0003842062724288553,
+                            0.0003835909010376781,
+                            0.0005568026099354029,
+                            0.0003850900975521654,
+                            0.0005921465926803648,
+                            0.0007004310609772801,
+                            0.0008674294222146273,
+                            0.0005073987995274365,
+                            0.0003857178380712867,
+                            0.0012163108913227916,
+                            0.0010327121708542109,
+                            0.0003831016074400395,
+                            0.0007276161923073232,
+                            0.00038241592119447887,
+                            0.0011224522022530437,
+                            0.0004145228595007211,
+                            0.0006604948430322111,
+                            0.0005839735385961831,
+                            0.00038561649853363633,
+                            0.0009218704071827233,
+                            0.0005145570612512529,
+                            0.0011110070627182722,
+                            0.0006232051528058946,
+                            0.0003842590085696429,
+                            0.0003844137245323509,
+                            0.0008797662449069321,
+                            0.0007112243911251426,
+                            0.0011161437723785639,
+                            0.0004231222555972636,
+                            0.00038543096161447465,
+                            0.0007002141792327166,
+                            0.0004984891274943948,
+                            0.0006912487442605197,
+                            0.0004269953060429543,
+                            0.0007799676386639476,
+                            0.00114572758320719,
+                            0.0004742721503134817,
+                            0.000896722951438278,
+                            0.0006188569241203368,
+                            0.0008833526517264545,
+                            0.0004168768064118922,
+                            0.0007567955181002617,
+                            0.00038393071736209095,
+                            0.00041881881770677865,
+                            0.0009783044224604964,
+                            0.0006782354321330786,
+                            0.00034830954973585904,
+                            0.0010672769276425242,
+                            0.00040633746539242566,
+                            0.00038292614044621587,
+                            0.0003837964031845331,
+                            0.0004235178348608315,
+                            0.0005030817119404674,
+                            0.0004198399547021836,
+                            0.0007976937922649086,
+                            0.00038426960236392915,
+                            0.000459088507341221,
+                            0.0007120526861399412,
+                            0.0003811428905464709,
+                            0.0003823197039309889,
+                            0.0008616486447863281,
+                            0.0005939012044109404,
+                            0.0006224765093065798,
+                            0.0005884924321435392,
+                            0.0008378751808777452,
+                            0.0003864902537316084,
+                            0.0011377987684682012,
+                            0.0003939637099392712,
+                            0.001204372732900083,
+                            0.0004360325401648879
+                        ],
+                        "zero_point": [
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0,
+                            0
+                        ]
+                    }
+                ],
+                "output_tensor_infos": [
+                    {
+                        "id": 78,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "scale": [
+                            0.040473222732543945
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            },
+            "1": {
+                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type_is_module": false,
+                "fqn": "over_arch.model.0._mlp.3",
+                "input_tensor_infos": [
+                    {
+                        "id": 78,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
+                        "scale": [
+                            0.040473222732543945
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "weight_tensor_infos": [],
+                "output_tensor_infos": [
+                    {
+                        "id": 79,
+                        "orig_dtype": "torch.float32",
+                        "inf_dtype": "torch.qint8",
+                        "scale": [
+                            0.04151982069015503
+                        ],
+                        "zero_point": [
+                            0
+                        ]
+                    }
+                ],
+                "activation_observer": {
+                    "name": "HistogramObserver",
+                    "bins": 127,
+                    "upsample_rate": 256,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_tensor_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -127,
+                    "quant_max": 126
+                },
+                "weight_observer": {
+                    "name": "PerChannelMinMaxObserver",
+                    "ch_axis": 0,
+                    "dtype": "torch.qint8",
+                    "qscheme": "torch.per_channel_symmetric",
+                    "reduce_range": false,
+                    "quant_min": -128,
+                    "quant_max": 127
+                }
+            }
+        },
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 79,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8",
+                "scale": [
+                    0.04151982069015503
+                ],
+                "zero_point": [
+                    0
+                ]
+            }
+        ]
+    }
+}
\ No newline at end of file
diff --git a/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py b/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py
new file mode 100644
index 000000000..698432290
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py
@@ -0,0 +1,199 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+
+from torchrec.models.dlrm import SparseArch, InteractionDCNArch, DLRM_DCN
+from torchrec.modules.embedding_modules import EmbeddingBagCollection
+
+from typing import Dict, List, Optional, Tuple
+import intel_extension_for_pytorch as ipex
+from intel_extension_for_pytorch.nn.modules import MergedEmbeddingBagWithCat
+
+class _LowRankCrossNet(torch.nn.Module):
+    def __init__(
+        self,
+        lr_crossnet,
+    ) -> None:
+        super().__init__()
+        self._num_layers = lr_crossnet._num_layers
+        self._in_features = lr_crossnet.bias[0].shape[0]
+        self._low_rank = lr_crossnet._low_rank
+        self.V_linears = torch.nn.ModuleList()
+        self.W_linears = torch.nn.ModuleList()
+        for i in range(self._num_layers):
+            self.V_linears.append(torch.nn.Linear(self._in_features, self._low_rank, bias=False))
+            self.W_linears.append(torch.nn.Linear(self._low_rank, self._in_features, bias=True))
+            self.V_linears[i].weight.data = lr_crossnet.V_kernels[i]
+            self.W_linears[i].weight.data = lr_crossnet.W_kernels[i]
+            self.W_linears[i].bias.data = lr_crossnet.bias[i]
+
+    def forward(self, input: torch.Tensor) -> torch.Tensor:
+        x_0 = input
+        x_l = x_0
+        for layer in range(self._num_layers):
+            x_l_v = self.V_linears[layer](x_l)
+            x_l_w = self.W_linears[layer](x_l_v)
+            x_l = x_0 * x_l_w + x_l  # (B, N)
+        return x_l
+
+def replace_crossnet(dlrm):
+    crossnet = dlrm.inter_arch.crossnet
+    new_crossnet = _LowRankCrossNet(crossnet)
+    dlrm.inter_arch = new_crossnet
+    del(crossnet)
+
+class SparseArchCatDense(SparseArch):
+
+    def forward(
+        self,
+        embedded_dense_features,
+        sparse_features,
+    ) -> torch.Tensor:
+        """
+        Args:
+            embedded_dense_features: the output of DenseArch.
+            sparse_features: the indices/offsets for F embeddingbags in embedding_bag_collection
+
+        Returns:
+            torch.Tensor: tensor of shape B X (F + 1) X D.
+        """
+        embedding_bag_collection = self.embedding_bag_collection
+        if isinstance(embedding_bag_collection, MergedEmbeddingBagWithCat):
+            indices = tuple([sf['values'] for _, sf in sparse_features.items()])
+            offsets = tuple([sf['offsets'] for _, sf in sparse_features.items()])
+            return embedding_bag_collection(indices, offsets, embedded_dense_features)
+        else:
+            embedded_sparse_features: List[torch.Tensor] = []
+            for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+                for feature_name in embedding_bag_collection._feature_names[i]:
+                    f = sparse_features[feature_name]
+                    res = embedding_bag(
+                        f['values'],
+                        f['offsets'],
+                        per_sample_weights=None,
+                    )
+                    embedded_sparse_features.append(res)
+            to_cat = [embedded_dense_features] + embedded_sparse_features
+            return torch.cat(to_cat, dim=1)
+
+def replace_embeddingbag_collection(dlrm):
+    collection = list(dlrm.sparse_arch.embedding_bag_collection.embedding_bags.values())
+    new_collection = MergedEmbeddingBagWithCat.from_embeddingbag_list(collection)
+    dlrm.sparse_arch.embedding_bag_collection = new_collection
+    del(collection)
+
+class InteractionDCNArchWithoutCat(InteractionDCNArch):
+
+    def forward(
+        self, concat_dense_sparse: torch.Tensor
+    ) -> torch.Tensor:
+        """
+        Args:
+            concat_dense_sparse (torch.Tensor): an input tensor of size B X (F*D + D).
+
+        Returns:
+            torch.Tensor: an output tensor of size B X (F*D + D).
+        """
+
+        # size B X (F * D + D)
+        return self.crossnet(concat_dense_sparse)
+
+class IPEX_DLRM_DCN(DLRM_DCN):
+    """
+    Recsys model with DCN modified from the original model from "Deep Learning Recommendation
+    Model for Personalization and Recommendation Systems"
+    (https://arxiv.org/abs/1906.00091). Similar to DLRM module but has
+    DeepCrossNet https://arxiv.org/pdf/2008.13535.pdf as the interaction layer.
+
+    The module assumes all sparse features have the same embedding dimension
+    (i.e. each EmbeddingBagConfig uses the same embedding_dim).
+
+    The following notation is used throughout the documentation for the models:
+
+    * F: number of sparse features
+    * D: embedding_dimension of sparse features
+    * B: batch size
+    * num_features: number of dense features
+
+    Args:
+        embedding_bag_collection (EmbeddingBagCollection): collection of embedding bags
+            used to define `SparseArch`.
+        dense_in_features (int): the dimensionality of the dense input features.
+        dense_arch_layer_sizes (List[int]): the layer sizes for the `DenseArch`.
+        over_arch_layer_sizes (List[int]): the layer sizes for the `OverArch`.
+            The output dimension of the `InteractionArch` should not be manually
+            specified here.
+        dcn_num_layers (int): the number of DCN layers in the interaction.
+        dcn_low_rank_dim (int): the dimensionality of low rank approximation
+            used in the dcn layers.
+        dense_device (Optional[torch.device]): default compute device.
+    """
+
+    def __init__(
+        self,
+        embedding_bag_collection: EmbeddingBagCollection,
+        dense_in_features: int,
+        dense_arch_layer_sizes: List[int],
+        over_arch_layer_sizes: List[int],
+        dcn_num_layers: int,
+        dcn_low_rank_dim: int,
+        dense_device: Optional[torch.device] = None,
+    ) -> None:
+        # initialize DLRM
+        # sparse arch and dense arch are initialized via DLRM
+        super().__init__(
+            embedding_bag_collection,
+            dense_in_features,
+            dense_arch_layer_sizes,
+            over_arch_layer_sizes,
+            dcn_num_layers,
+            dcn_low_rank_dim,
+            dense_device,
+        )
+
+        num_sparse_features: int = len(self.sparse_arch.sparse_feature_names)
+
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        self.sparse_arch = SparseArchCatDense(embedding_bag_collection)
+
+        crossnet = self.inter_arch.crossnet
+        self.inter_arch = InteractionDCNArchWithoutCat(
+            num_sparse_features=num_sparse_features,
+            crossnet=crossnet,
+        )
+
+    def forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_dense = self.dense_arch(dense_features)
+        concat_sparse_dense = self.sparse_arch(embedded_dense, sparse_features)
+        concatenated_dense = self.inter_arch(concat_sparse_dense)
+        logits = self.over_arch(concatenated_dense)
+        return logits
diff --git a/models/recommendation/pytorch/torchrec_dlrm/jit_trace_able_utils.py b/models/recommendation/pytorch/torchrec_dlrm/jit_trace_able_utils.py
new file mode 100644
index 000000000..36dd164a1
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/jit_trace_able_utils.py
@@ -0,0 +1,63 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+from torchrec.models.dlrm import SparseArch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from typing import List
+
+def unpack(input :KeyedJaggedTensor) -> dict:
+    output = {}
+    for k, v in input.to_dict().items():
+        output[k] = {}
+        output[k]['values'] = v._values.int()
+        output[k]['offsets'] = v._offsets.int()
+    return output
+
+class SparseArchTraceAbleWrapper(nn.Module):
+
+    def __init__(self, sparse_arch: SparseArch) -> None:
+        super().__init__()
+        self.sparse_arch: SparseArch = sparse_arch
+
+    def forward(
+        self,
+        features: dict,
+    ) -> torch.Tensor:
+        """
+        Args:
+            features (dict): input tensors of sparse features.
+
+        Returns:
+            torch.Tensor: tensor of shape B X F X D.
+        """
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        pooled_embeddings: List[torch.Tensor] = []
+        for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+            for feature_name in embedding_bag_collection._feature_names[i]:
+                f = features[feature_name]
+                res = embedding_bag(
+                    f['values'],
+                    f['offsets'],
+                    per_sample_weights=None,
+                ).float()
+                pooled_embeddings.append(res)
+        data = torch.cat(pooled_embeddings, dim=1)
+        B: int = features['cat_0']['offsets'].numel() - 1
+        return data.reshape(B, self.sparse_arch.F, self.sparse_arch.D)
\ No newline at end of file
diff --git a/models/recommendation/pytorch/torchrec_dlrm/lr_scheduler.py b/models/recommendation/pytorch/torchrec_dlrm/lr_scheduler.py
new file mode 100644
index 000000000..4aebce416
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/lr_scheduler.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Copied from https://github.com/facebookresearch/dlrm/blob/mlperf/dlrm_s_pytorch.py
+
+import sys
+
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
diff --git a/models/recommendation/pytorch/torchrec_dlrm/multi_hot.py b/models/recommendation/pytorch/torchrec_dlrm/multi_hot.py
new file mode 100644
index 000000000..b66f80ad1
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/multi_hot.py
@@ -0,0 +1,191 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import List, Tuple
+
+import numpy as np
+
+import torch
+from torchrec.datasets.utils import Batch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+import gc
+
+class RestartableMap:
+    def __init__(self, f, source):
+        self.source = source
+        self.func = f
+
+    def __iter__(self):
+        for x in self.source:
+            yield self.func(x)
+
+    def __len__(self):
+        return len(self.source)
+
+
+class Multihot:
+    def __init__(
+        self,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+        collect_freqs_stats: bool,
+        dist_type: str = "uniform",
+    ):
+        if dist_type not in {"uniform", "pareto"}:
+            raise ValueError(
+                "Multi-hot distribution type {} is not supported."
+                'Only "uniform" and "pareto" are supported.'.format(dist_type)
+            )
+        self.dist_type = dist_type
+        self.multi_hot_sizes = multi_hot_sizes
+        self.num_embeddings_per_feature = num_embeddings_per_feature
+        self.batch_size = batch_size
+
+        # Generate 1-hot to multi-hot lookup tables, one lookup table per sparse embedding table.
+        self.multi_hot_tables_l = self.__make_multi_hot_indices_tables(
+            dist_type, multi_hot_sizes, num_embeddings_per_feature
+        )
+
+        # Pooling offsets are computed once and reused.
+        self.offsets = self.__make_offsets(
+            multi_hot_sizes, num_embeddings_per_feature, batch_size
+        )
+
+        # For plotting frequency access
+        self.collect_freqs_stats = collect_freqs_stats
+        self.model_to_track = None
+        self.freqs_pre_hash = []
+        self.freqs_post_hash = []
+        for embs_count in num_embeddings_per_feature:
+            self.freqs_pre_hash.append(np.zeros((embs_count)))
+            self.freqs_post_hash.append(np.zeros((embs_count)))
+
+    def save_freqs_stats(self) -> None:
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            rank = torch.distributed.get_rank()
+        else:
+            rank = 0
+        pre_dict = {str(k): e for k, e in enumerate(self.freqs_pre_hash)}
+        np.save(f"stats_pre_hash_{rank}_{self.dist_type}.npy", pre_dict)
+        post_dict = {str(k): e for k, e in enumerate(self.freqs_post_hash)}
+        np.save(f"stats_post_hash_{rank}_{self.dist_type}.npy", post_dict)
+
+    def pause_stats_collection_during_val_and_test(
+        self, model: torch.nn.Module
+    ) -> None:
+        self.model_to_track = model
+
+    def __make_multi_hot_indices_tables(
+        self,
+        dist_type: str,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+    ) -> List[np.array]:
+        np.random.seed(
+            0
+        )  # The seed is necessary for all ranks to produce the same lookup values.
+        multi_hot_tables_l = []
+        for embs_count, multi_hot_size in zip(
+            num_embeddings_per_feature, multi_hot_sizes
+        ):
+            multi_hot_table = torch.empty((embs_count, multi_hot_size), dtype=torch.int)
+            multi_hot_table[:, 0].copy_(torch.arange(embs_count, dtype=torch.int))
+            if dist_type == "uniform":
+                synthetic_sparse_ids = np.random.randint(
+                    0, embs_count, size=(embs_count, multi_hot_size - 1), dtype=np.int32
+                )
+            elif dist_type == "pareto":
+                synthetic_sparse_ids = (
+                    np.random.pareto(
+                        a=0.25, size=(embs_count, multi_hot_size - 1)
+                    ).astype(np.int32)
+                    % embs_count
+                )
+            synthetic_sparse_ids = torch.from_numpy(synthetic_sparse_ids).int()
+            multi_hot_table[:, 1:].copy_(synthetic_sparse_ids)
+            del synthetic_sparse_ids
+            gc.collect()
+            multi_hot_tables_l.append(multi_hot_table)
+        return multi_hot_tables_l
+
+    def __make_offsets(
+        self,
+        multi_hot_sizes: int,
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+    ) -> List[torch.Tensor]:
+        lS_o = torch.ones(
+            (len(num_embeddings_per_feature) * batch_size), dtype=torch.int32
+        )
+        for k, multi_hot_size in enumerate(multi_hot_sizes):
+            lS_o[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        lS_o = torch.cumsum(torch.concat((torch.tensor([0]), lS_o)), axis=0)
+        return lS_o
+
+    def __make_new_batch(
+        self,
+        lS_i: torch.Tensor,
+        batch_size: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        lS_i = lS_i.reshape(-1, batch_size)
+        multi_hot_ids_l = []
+        for k, (sparse_data_batch_for_table, multi_hot_table) in enumerate(
+            zip(lS_i, self.multi_hot_tables_l)
+        ):
+            multi_hot_ids = torch.nn.functional.embedding(
+                sparse_data_batch_for_table, multi_hot_table
+            )
+            multi_hot_ids = multi_hot_ids.reshape(-1)
+            multi_hot_ids_l.append(multi_hot_ids)
+            if self.collect_freqs_stats and (
+                self.model_to_track is None or self.model_to_track.training
+            ):
+                idx_pre, cnt_pre = np.unique(
+                    sparse_data_batch_for_table, return_counts=True
+                )
+                idx_post, cnt_post = np.unique(multi_hot_ids, return_counts=True)
+                self.freqs_pre_hash[k][idx_pre] += cnt_pre
+                self.freqs_post_hash[k][idx_post] += cnt_post
+        lS_i = torch.cat(multi_hot_ids_l)
+        if batch_size == self.batch_size:
+            return lS_i, self.offsets
+        else:
+            return lS_i, self.__make_offsets(
+                self.multi_hot_sizes, self.num_embeddings_per_feature, batch_size
+            )
+
+    def convert_to_multi_hot(self, batch: Batch) -> Batch:
+        batch_size = len(batch.dense_features)
+        lS_i = batch.sparse_features._values
+        lS_i, lS_o = self.__make_new_batch(lS_i, batch_size)
+        new_sparse_features = KeyedJaggedTensor.from_offsets_sync(
+            keys=batch.sparse_features._keys,
+            values=lS_i,
+            offsets=lS_o,
+        )
+        return Batch(
+            dense_features=batch.dense_features,
+            sparse_features=new_sparse_features,
+            labels=batch.labels,
+        )
diff --git a/models/recommendation/pytorch/torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py b/models/recommendation/pytorch/torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py
new file mode 100644
index 000000000..ec7562258
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py
@@ -0,0 +1,171 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+import pathlib
+import shutil
+import sys
+
+import numpy as np
+import torch
+from torch import distributed as dist, nn
+from torchrec.datasets.criteo import DAYS
+
+p = pathlib.Path(__file__).absolute().parents[1].resolve()
+sys.path.append(os.fspath(p))
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot import Multihot  # noqa F811
+except ImportError:
+    pass
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Script to materialize synthetic multi-hot dataset into NumPy npz file format."
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        required=True,
+        help="Path to a folder containing the binary (npy) files for the Criteo dataset."
+        " When supplied, InMemoryBinaryCriteoIterDataPipe is used.",
+    )
+    parser.add_argument(
+        "--output_path",
+        type=str,
+        required=True,
+        help="Path to outputted multi-hot sparse dataset.",
+    )
+    parser.add_argument(
+        "--copy_labels_and_dense",
+        dest="copy_labels_and_dense",
+        action="store_true",
+        help="Flag to determine whether to copy labels and dense data to the output directory.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        required=True,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        required=True,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default="uniform",
+        help="Multi-hot distribution options.",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    """
+    This script generates and saves the MLPerf v2 multi-hot dataset (4 TB in size).
+    First, run process_Criteo_1TB_Click_Logs_dataset.sh.
+    Then, run this script as follows:
+
+        python materialize_synthetic_multihot_dataset.py \
+            --in_memory_binary_criteo_path $PREPROCESSED_CRITEO_1TB_CLICK_LOGS_DATASET_PATH \
+            --output_path $MATERIALIZED_DATASET_PATH \
+            --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+            --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+            --multi_hot_distribution_type uniform
+
+    This script takes about 2 hours to run (can be parallelized if needed).
+    """
+    args = parse_args()
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    try:
+        backend = "nccl" if torch.cuda.is_available() else "gloo"
+        if not dist.is_initialized():
+            dist.init_process_group(backend=backend)
+        rank = dist.get_rank()
+        world_size = dist.get_world_size()
+    except (KeyError, ValueError):
+        rank = 0
+        world_size = 1
+
+    print("Generating one-hot to multi-hot lookup table.")
+    multihot = Multihot(
+        multi_hot_sizes=args.multi_hot_sizes,
+        num_embeddings_per_feature=args.num_embeddings_per_feature,
+        batch_size=1,  # Doesn't matter
+        collect_freqs_stats=False,
+        dist_type=args.multi_hot_distribution_type,
+    )
+
+    os.makedirs(args.output_path, exist_ok=True)
+
+    for i in range(rank, DAYS, world_size):
+        input_file_path = os.path.join(
+            args.in_memory_binary_criteo_path, f"day_{i}_sparse.npy"
+        )
+        print(f"Materializing {input_file_path}")
+        sparse_data = np.load(input_file_path, mmap_mode="r")
+        multi_hot_ids_dict = {}
+        for j, (multi_hot_table, hash) in enumerate(
+            zip(multihot.multi_hot_tables_l, args.num_embeddings_per_feature)
+        ):
+            sparse_tensor = torch.from_numpy(sparse_data[:, j] % hash)
+            multi_hot_ids_dict[str(j)] = nn.functional.embedding(
+                sparse_tensor, multi_hot_table
+            ).numpy()
+        output_file_path = os.path.join(
+            args.output_path, f"day_{i}_sparse_multi_hot.npz"
+        )
+        np.savez(output_file_path, **multi_hot_ids_dict)
+        if args.copy_labels_and_dense:
+            for part in ["labels", "dense"]:
+                source_path = os.path.join(
+                    args.in_memory_binary_criteo_path, f"day_{i}_{part}.npy"
+                )
+                output_path = os.path.join(args.output_path, f"day_{i}_{part}.npy")
+                shutil.copyfile(source_path, output_path)
+                print(f"Copying {source_path} to {output_path}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models/recommendation/pytorch/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh b/models/recommendation/pytorch/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
new file mode 100644
index 000000000..23050647d
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+display_help() {
+   echo "Three command line arguments are required."
+   echo "Example usage:"
+   echo "bash process_Criteo_1TB_Click_Logs_dataset.sh \\"
+   echo "./criteo_1tb/raw_input_dataset_dir \\"
+   echo "./criteo_1tb/temp_intermediate_files_dir \\"
+   echo "./criteo_1tb/numpy_contiguous_shuffled_output_dataset_dir"
+   exit 1
+}
+
+[ -z "$1" ] && display_help
+[ -z "$2" ] && display_help
+[ -z "$3" ] && display_help
+
+# Input directory containing the raw Criteo 1TB Click Logs dataset files in tsv format.
+# The 24 dataset filenames in the directory should be day_{0..23} with no .tsv extension.
+raw_tsv_criteo_files_dir=$(readlink -m "$1")
+
+# Directory to store temporary intermediate output files created by preprocessing steps 1 and 2.
+temp_files_dir=$(readlink -m "$2")
+
+# Directory to store temporary intermediate output files created by preprocessing step 1.
+step_1_output_dir="$temp_files_dir/temp_output_of_step_1"
+
+# Directory to store temporary intermediate output files created by preprocessing step 2.
+step_2_output_dir="$temp_files_dir/temp_output_of_step_2"
+
+# Directory to store the final preprocessed Criteo 1TB Click Logs dataset.
+step_3_output_dir=$(readlink -m "$3")
+
+# Step 1. Split the dataset into 3 sets of 24 numpy files:
+# day_{0..23}_dense.npy, day_{0..23}_labels.npy, and day_{0..23}_sparse.npy (~24hrs)
+set -x
+mkdir -p "$step_1_output_dir"
+date
+python -m torchrec.datasets.scripts.npy_preproc_criteo --input_dir "$raw_tsv_criteo_files_dir" --output_dir "$step_1_output_dir" || exit
+
+# Step 2. Convert all sparse indices in day_{0..23}_sparse.npy to contiguous indices and save the output.
+# The output filenames are day_{0..23}_sparse_contig_freq.npy
+mkdir -p "$step_2_output_dir"
+date
+python -m torchrec.datasets.scripts.contiguous_preproc_criteo --input_dir "$step_1_output_dir" --output_dir "$step_2_output_dir" --frequency_threshold 0 || exit
+
+date
+for i in {0..23}
+do
+   name="$step_2_output_dir/day_$i""_sparse_contig_freq.npy"
+   renamed="$step_2_output_dir/day_$i""_sparse.npy"
+   echo "Renaming $name to $renamed"
+   mv "$name" "$renamed"
+done
+
+# Step 3. Shuffle the dataset's samples in days 0 through 22. (~20hrs)
+# Day 23's samples are not shuffled and will be used for the validation set and test set.
+mkdir -p "$step_3_output_dir"
+date
+python -m torchrec.datasets.scripts.shuffle_preproc_criteo --input_dir_labels_and_dense "$step_1_output_dir" --input_dir_sparse "$step_2_output_dir" --output_dir_shuffled "$step_3_output_dir" --random_seed 0 || exit
+date
diff --git a/models/image_recognition/tensorflow/vision_transformer/training/__init__.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/__init__.py
similarity index 94%
rename from models/image_recognition/tensorflow/vision_transformer/training/__init__.py
rename to models/recommendation/pytorch/torchrec_dlrm/training/gpu/__init__.py
index f07dbaf88..994d8d5cb 100644
--- a/models/image_recognition/tensorflow/vision_transformer/training/__init__.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/__init__.py
@@ -1,6 +1,4 @@
-#
-# -*- coding: utf-8 -*-
-#
+#!/bin/bash
 # Copyright (c) 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,5 +13,3 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-
-#
diff --git a/models/image_recognition/tensorflow/vision_transformer/__init__.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/__init__.py
similarity index 94%
rename from models/image_recognition/tensorflow/vision_transformer/__init__.py
rename to models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/__init__.py
index f07dbaf88..994d8d5cb 100644
--- a/models/image_recognition/tensorflow/vision_transformer/__init__.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/__init__.py
@@ -1,6 +1,4 @@
-#
-# -*- coding: utf-8 -*-
-#
+#!/bin/bash
 # Copyright (c) 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -15,5 +13,3 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-
-#
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py
new file mode 100644
index 000000000..384f8fe89
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+from typing import List
+
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    #DAYS,
+    DEFAULT_CAT_NAMES,
+    DEFAULT_INT_NAMES,
+    InMemoryBinaryCriteoIterDataPipe,
+)
+# This is for crop dataset
+DAYS=1
+from torchrec.datasets.random import RandomRecDataset
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:multi_hot_criteo
+    from data.multi_hot_criteo import MultiHotCriteoIterDataPipe
+
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot_criteo import MultiHotCriteoIterDataPipe  # noqa F811
+except ImportError:
+    pass
+
+STAGES = ["train", "val", "test"]
+
+
+def _get_random_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    attr = f"limit_{stage}_batches"
+    num_batches = getattr(args, attr)
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    return DataLoader(
+        RandomRecDataset(
+            keys=DEFAULT_CAT_NAMES,
+            batch_size=batch_size,
+            hash_size=args.num_embeddings,
+            hash_sizes=args.num_embeddings_per_feature
+            if hasattr(args, "num_embeddings_per_feature")
+            else None,
+            manual_seed=args.seed if hasattr(args, "seed") else None,
+            ids_per_feature=1,
+            num_dense=len(DEFAULT_INT_NAMES),
+            num_batches=num_batches,
+        ),
+        batch_size=None,
+        batch_sampler=None,
+        pin_memory=args.pin_memory,
+        num_workers=0,
+    )
+
+
+def _get_in_memory_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    if args.in_memory_binary_criteo_path is not None:
+        dir_path = args.in_memory_binary_criteo_path
+        sparse_part = "sparse.npy"
+        datapipe = InMemoryBinaryCriteoIterDataPipe
+    else:
+        dir_path = args.synthetic_multi_hot_criteo_path
+        sparse_part = "sparse_multi_hot.npz"
+        datapipe = MultiHotCriteoIterDataPipe
+
+    if args.dataset_name == "criteo_kaggle":
+        # criteo_kaggle has no validation set, so use 2nd half of training set for now.
+        # Setting stage to "test" will get the 2nd half of the dataset.
+        # Setting root_name to "train" reads from the training set file.
+        (root_name, stage) = ("train", "test") if stage == "val" else stage
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"{root_name}_dense.npy")],
+            [os.path.join(dir_path, f"{root_name}_{sparse_part}")],
+            [os.path.join(dir_path, f"{root_name}_labels.npy")],
+        ]
+    # criteo_1tb code path uses below two conditionals
+    elif stage == "train":
+        stage_files: List[List[str]] = [
+            #[os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS - 1)],
+            #[os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS - 1)],
+            #[os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS - 1)],
+            # for crop dataset
+            [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS)],
+            [os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS)],
+            [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS)],
+        ]
+    elif stage in ["val", "test"]:
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"day_{DAYS-1}_dense.npy")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_{sparse_part}")],
+            [os.path.join(dir_path, f"day_{DAYS-1}_labels.npy")],
+        ]
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    print(stage_files)
+    dataloader = DataLoader(
+        datapipe(
+            stage,
+            *stage_files,  # pyre-ignore[6]
+            batch_size=batch_size,
+            #rank=dist.get_rank(),
+            #world_size=dist.get_world_size(),
+            # The rand and world_size set for custom dist-dlrm
+            rank=0,
+            world_size=1,
+            drop_last=args.drop_last_training_batch if stage == "train" else False,
+            shuffle_batches=args.shuffle_batches,
+            shuffle_training_set=args.shuffle_training_set,
+            shuffle_training_set_random_seed=args.seed,
+            mmap_mode=args.mmap_mode,
+            hashes=args.num_embeddings_per_feature
+            if args.num_embeddings is None
+            else ([args.num_embeddings] * CAT_FEATURE_COUNT),
+        ),
+        batch_size=None,
+        pin_memory=args.pin_memory,
+        collate_fn=lambda x: x,
+    )
+    return dataloader
+
+
+def get_dataloader(args: argparse.Namespace, backend: str, stage: str) -> DataLoader:
+    """
+    Gets desired dataloader from dlrm_main command line options. Currently, this
+    function is able to return either a DataLoader wrapped around a RandomRecDataset or
+    a Dataloader wrapped around an InMemoryBinaryCriteoIterDataPipe.
+
+    Args:
+        args (argparse.Namespace): Command line options supplied to dlrm_main.py's main
+            function.
+        backend (str): "nccl" or "gloo".
+        stage (str): "train", "val", or "test".
+
+    Returns:
+        dataloader (DataLoader): PyTorch dataloader for the specified options.
+
+    """
+    stage = stage.lower()
+    if stage not in STAGES:
+        raise ValueError(f"Supplied stage was {stage}. Must be one of {STAGES}.")
+
+    args.pin_memory = (
+        (backend == "nccl") if not hasattr(args, "pin_memory") else args.pin_memory
+    )
+
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        return _get_random_dataloader(args, stage)
+    else:
+        return _get_in_memory_dataloader(args, stage)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py
new file mode 100644
index 000000000..cb520e2ec
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py
@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import zipfile
+from typing import Dict, Iterator, List, Optional
+
+import numpy as np
+import torch
+from iopath.common.file_io import PathManager, PathManagerFactory
+from pyre_extensions import none_throws
+from torch.utils.data import IterableDataset
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DEFAULT_CAT_NAMES,
+)
+from torchrec.datasets.utils import Batch, PATH_MANAGER_KEY
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class MultiHotCriteoIterDataPipe(IterableDataset):
+    """
+    Datapipe designed to operate over the MLPerf DLRM v2 synthetic multi-hot dataset.
+    This dataset can be created by following the steps in
+    torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py.
+    Each rank reads only the data for the portion of the dataset it is responsible for.
+
+    Args:
+        stage (str): "train", "val", or "test".
+        dense_paths (List[str]): List of path strings to dense npy files.
+        sparse_paths (List[str]): List of path strings to multi-hot sparse npz files.
+        labels_paths (List[str]): List of path strings to labels npy files.
+        batch_size (int): batch size.
+        rank (int): rank.
+        world_size (int): world size.
+        drop_last (Optional[bool]): Whether to drop the last batch if it is incomplete.
+        shuffle_batches (bool): Whether to shuffle batches
+        shuffle_training_set (bool): Whether to shuffle all samples in the dataset.
+        shuffle_training_set_random_seed (int): The random generator seed used when
+            shuffling the training set.
+        hashes (Optional[int]): List of max categorical feature value for each feature.
+            Length of this list should be CAT_FEATURE_COUNT.
+        path_manager_key (str): Path manager key used to load from different
+            filesystems.
+
+    Example::
+
+        datapipe = MultiHotCriteoIterDataPipe(
+            dense_paths=["day_0_dense.npy"],
+            sparse_paths=["day_0_sparse_multi_hot.npz"],
+            labels_paths=["day_0_labels.npy"],
+            batch_size=1024,
+            rank=torch.distributed.get_rank(),
+            world_size=torch.distributed.get_world_size(),
+        )
+        batch = next(iter(datapipe))
+    """
+
+    def __init__(
+        self,
+        stage: str,
+        dense_paths: List[str],
+        sparse_paths: List[str],
+        labels_paths: List[str],
+        batch_size: int,
+        rank: int,
+        world_size: int,
+        drop_last: Optional[bool] = False,
+        shuffle_batches: bool = False,
+        shuffle_training_set: bool = False,
+        shuffle_training_set_random_seed: int = 0,
+        mmap_mode: bool = False,
+        hashes: Optional[List[int]] = None,
+        path_manager_key: str = PATH_MANAGER_KEY,
+    ) -> None:
+        self.stage = stage
+        self.dense_paths = dense_paths
+        self.sparse_paths = sparse_paths
+        self.labels_paths = labels_paths
+        self.batch_size = batch_size
+        self.rank = rank
+        self.world_size = world_size
+        self.drop_last = drop_last
+        self.shuffle_batches = shuffle_batches
+        self.shuffle_training_set = shuffle_training_set
+        np.random.seed(shuffle_training_set_random_seed)
+        self.mmap_mode = mmap_mode
+        # hashes are not used because they were already applied in the
+        # script that generates the multi-hot dataset.
+        self.hashes: np.ndarray = np.array(hashes).reshape((CAT_FEATURE_COUNT, 1))
+        self.path_manager_key = path_manager_key
+        self.path_manager: PathManager = PathManagerFactory().get(path_manager_key)
+
+        if shuffle_training_set and stage == "train":
+            # Currently not implemented for the materialized multi-hot dataset.
+            self._shuffle_and_load_data_for_rank()
+        else:
+            m = "r" if mmap_mode else None
+            self.dense_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.dense_paths
+            ]
+            self.labels_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.labels_paths
+            ]
+            self.sparse_arrs: List = []
+            for sparse_path in self.sparse_paths:
+                multi_hot_ids_l = []
+                for feat_id_num in range(CAT_FEATURE_COUNT):
+                    multi_hot_ft_ids = self._load_from_npz(
+                        sparse_path, f"{feat_id_num}.npy"
+                    )
+                    multi_hot_ids_l.append(multi_hot_ft_ids)
+                self.sparse_arrs.append(multi_hot_ids_l)
+        len_d0 = len(self.dense_arrs[0])
+        second_half_start_index = int(len_d0 // 2 + len_d0 % 2)
+        if stage == "val":
+            self.dense_arrs[0] = self.dense_arrs[0][:second_half_start_index, :]
+            self.labels_arrs[0] = self.labels_arrs[0][:second_half_start_index, :]
+            self.sparse_arrs[0] = [
+                feats[:second_half_start_index, :] for feats in self.sparse_arrs[0]
+            ]
+        elif stage == "test":
+            self.dense_arrs[0] = self.dense_arrs[0][second_half_start_index:, :]
+            self.labels_arrs[0] = self.labels_arrs[0][second_half_start_index:, :]
+            self.sparse_arrs[0] = [
+                feats[second_half_start_index:, :] for feats in self.sparse_arrs[0]
+            ]
+        # When mmap_mode is enabled, sparse features are hashed when
+        # samples are batched in def __iter__. Otherwise, the dataset has been
+        # preloaded with sparse features hashed in the preload stage, here:
+        # if not self.mmap_mode and self.hashes is not None:
+        #     for k, _ in enumerate(self.sparse_arrs):
+        #         self.sparse_arrs[k] = [
+        #             feat % hash
+        #             for (feat, hash) in zip(self.sparse_arrs[k], self.hashes)
+        #         ]
+
+        self.num_rows_per_file: List[int] = list(map(len, self.dense_arrs))
+        total_rows = sum(self.num_rows_per_file)
+        self.num_full_batches: int = (
+            total_rows // batch_size // self.world_size * self.world_size
+        )
+        self.last_batch_sizes: np.ndarray = np.array(
+            [0 for _ in range(self.world_size)]
+        )
+        remainder = total_rows % (self.world_size * batch_size)
+        if not self.drop_last and 0 < remainder:
+            if remainder < self.world_size:
+                self.num_full_batches -= self.world_size
+                self.last_batch_sizes += batch_size
+            else:
+                self.last_batch_sizes += remainder // self.world_size
+            self.last_batch_sizes[: remainder % self.world_size] += 1
+
+        self.multi_hot_sizes: List[int] = [
+            multi_hot_feat.shape[-1] for multi_hot_feat in self.sparse_arrs[0]
+        ]
+
+        # These values are the same for the KeyedJaggedTensors in all batches, so they
+        # are computed once here. This avoids extra work from the KeyedJaggedTensor sync
+        # functions.
+        self.keys: List[str] = DEFAULT_CAT_NAMES
+        self.index_per_key: Dict[str, int] = {
+            key: i for (i, key) in enumerate(self.keys)
+        }
+
+    def _load_from_npz(self, fname, npy_name):
+        # figure out offset of .npy in .npz
+        zf = zipfile.ZipFile(fname)
+        info = zf.NameToInfo[npy_name]
+        assert info.compress_type == 0
+        zf.fp.seek(info.header_offset + len(info.FileHeader()) + 20)
+        # read .npy header
+        zf.open(npy_name, "r")
+        version = np.lib.format.read_magic(zf.fp)
+        shape, fortran_order, dtype = np.lib.format._read_array_header(zf.fp, version)
+        assert (
+            dtype == "int32"
+        ), f"sparse multi-hot dtype is {dtype} but should be int32"
+        offset = zf.fp.tell()
+        # create memmap
+        return np.memmap(
+            zf.filename,
+            dtype=dtype,
+            shape=shape,
+            order="F" if fortran_order else "C",
+            mode="r",
+            offset=offset,
+        )
+
+    def _np_arrays_to_batch(
+        self,
+        dense: np.ndarray,
+        sparse: List[np.ndarray],
+        labels: np.ndarray,
+    ) -> Batch:
+        if self.shuffle_batches:
+            # Shuffle all 3 in unison
+            shuffler = np.random.permutation(len(dense))
+            sparse = [multi_hot_ft[shuffler, :] for multi_hot_ft in sparse]
+            dense = dense[shuffler]
+            labels = labels[shuffler]
+
+        batch_size = len(dense)
+        lengths = torch.ones((CAT_FEATURE_COUNT * batch_size), dtype=torch.int32)
+        for k, multi_hot_size in enumerate(self.multi_hot_sizes):
+            lengths[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        offsets = torch.cumsum(torch.concat((torch.tensor([0]), lengths)), dim=0)
+        length_per_key = [
+            batch_size * multi_hot_size for multi_hot_size in self.multi_hot_sizes
+        ]
+        offset_per_key = torch.cumsum(
+            torch.concat((torch.tensor([0]), torch.tensor(length_per_key))), dim=0
+        )
+        values = torch.concat([torch.from_numpy(feat).flatten() for feat in sparse])
+        return Batch(
+            dense_features=torch.from_numpy(dense.copy()),
+            sparse_features=KeyedJaggedTensor(
+                keys=self.keys,
+                values=values,
+                lengths=lengths,
+                offsets=offsets,
+                stride=batch_size,
+                length_per_key=length_per_key,
+                offset_per_key=offset_per_key.tolist(),
+                index_per_key=self.index_per_key,
+            ),
+            labels=torch.from_numpy(labels.reshape(-1).copy()),
+        )
+
+    def __iter__(self) -> Iterator[Batch]:
+        # Invariant: buffer never contains more than batch_size rows.
+        buffer: Optional[List[np.ndarray]] = None
+
+        def append_to_buffer(
+            dense: np.ndarray,
+            sparse: List[np.ndarray],
+            labels: np.ndarray,
+        ) -> None:
+            nonlocal buffer
+            if buffer is None:
+                buffer = [dense, sparse, labels]
+            else:
+                buffer[0] = np.concatenate((buffer[0], dense))
+                buffer[1] = [np.concatenate((b, s)) for b, s in zip(buffer[1], sparse)]
+                buffer[2] = np.concatenate((buffer[2], labels))
+
+        # Maintain a buffer that can contain up to batch_size rows. Fill buffer as
+        # much as possible on each iteration. Only return a new batch when batch_size
+        # rows are filled.
+        file_idx = 0
+        row_idx = 0
+        batch_idx = 0
+        buffer_row_count = 0
+        cur_batch_size = (
+            self.batch_size if self.num_full_batches > 0 else self.last_batch_sizes[0]
+        )
+        while (
+            batch_idx
+            < self.num_full_batches + (self.last_batch_sizes[0] > 0) * self.world_size
+        ):
+            if buffer_row_count == cur_batch_size or file_idx == len(self.dense_arrs):
+                if batch_idx % self.world_size == self.rank:
+                    yield self._np_arrays_to_batch(*none_throws(buffer))
+                    buffer = None
+                buffer_row_count = 0
+                batch_idx += 1
+                if (
+                    0 <= batch_idx - self.num_full_batches < self.world_size
+                    and (self.last_batch_sizes[0] > 0)
+                ):
+                    cur_batch_size = self.last_batch_sizes[
+                        batch_idx - self.num_full_batches
+                    ]
+            else:
+                rows_to_get = min(
+                    cur_batch_size - buffer_row_count,
+                    self.num_rows_per_file[file_idx] - row_idx,
+                )
+                buffer_row_count += rows_to_get
+                slice_ = slice(row_idx, row_idx + rows_to_get)
+
+                if batch_idx % self.world_size == self.rank:
+                    dense_inputs = self.dense_arrs[file_idx][slice_, :]
+                    sparse_inputs = [
+                        feats[slice_, :] for feats in self.sparse_arrs[file_idx]
+                    ]
+                    target_labels = self.labels_arrs[file_idx][slice_, :]
+
+                    # if self.mmap_mode and self.hashes is not None:
+                    #     sparse_inputs = [
+                    #         feats % hash
+                    #         for (feats, hash) in zip(sparse_inputs, self.hashes)
+                    #     ]
+
+                    append_to_buffer(
+                        dense_inputs,
+                        sparse_inputs,
+                        target_labels,
+                    )
+                row_idx += rows_to_get
+
+                if row_idx >= self.num_rows_per_file[file_idx]:
+                    file_idx += 1
+                    row_idx = 0
+
+    def __len__(self) -> int:
+        return self.num_full_batches // self.world_size + (self.last_batch_sizes[0] > 0)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dist_models.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dist_models.py
new file mode 100644
index 000000000..eff746a8a
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dist_models.py
@@ -0,0 +1,241 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import torch
+from torchrec import EmbeddingBagCollection
+import extend_distributed as ext_dist
+from torchrec.models.dlrm import DLRM, InteractionDCNArch, OverArch
+from typing import Dict, List, Optional, Tuple
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from torchrec.modules.crossnet import LowRankCrossNet
+import time
+
+class DIST_DLRM_DCN(DLRM):
+    """
+    Recsys model with DCN modified from the original model from "Deep Learning Recommendation
+    Model for Personalization and Recommendation Systems"
+    (https://arxiv.org/abs/1906.00091). Similar to DLRM module but has
+    DeepCrossNet https://arxiv.org/pdf/2008.13535.pdf as the interaction layer.
+
+    The module assumes all sparse features have the same embedding dimension
+    (i.e. each EmbeddingBagConfig uses the same embedding_dim).
+
+    The following notation is used throughout the documentation for the models:
+
+    * F: number of sparse features
+    * D: embedding_dimension of sparse features
+    * B: batch size
+    * num_features: number of dense features
+
+    Args:
+        embedding_bag_collection (EmbeddingBagCollection): collection of embedding bags
+            used to define `SparseArch`.
+        dense_in_features (int): the dimensionality of the dense input features.
+        dense_arch_layer_sizes (List[int]): the layer sizes for the `DenseArch`.
+        over_arch_layer_sizes (List[int]): the layer sizes for the `OverArch`.
+            The output dimension of the `InteractionArch` should not be manually
+            specified here.
+        dcn_num_layers (int): the number of DCN layers in the interaction.
+        dcn_low_rank_dim (int): the dimensionality of low rank approximation
+            used in the dcn layers.
+        dense_device (Optional[torch.device]): default compute device.
+
+    Example::
+
+        B = 2
+        D = 8
+
+        eb1_config = EmbeddingBagConfig(
+           name="t1", embedding_dim=D, num_embeddings=100, feature_names=["f1", "f3"]
+        )
+        eb2_config = EmbeddingBagConfig(
+           name="t2",
+           embedding_dim=D,
+           num_embeddings=100,
+           feature_names=["f2"],
+        )
+
+        ebc = EmbeddingBagCollection(tables=[eb1_config, eb2_config])
+        model = DLRM_DCN(
+           embedding_bag_collection=ebc,
+           dense_in_features=100,
+           dense_arch_layer_sizes=[20, D],
+           dcn_num_layers=2,
+           dcn_low_rank_dim=8,
+           over_arch_layer_sizes=[5, 1],
+        )
+
+        features = torch.rand((B, 100))
+
+        #     0       1
+        # 0   [1,2] [4,5]
+        # 1   [4,3] [2,9]
+        # ^
+        # feature
+        sparse_features = KeyedJaggedTensor.from_offsets_sync(
+           keys=["f1", "f3"],
+           values=torch.tensor([1, 2, 4, 5, 4, 3, 2, 9]),
+           offsets=torch.tensor([0, 2, 4, 6, 8]),
+        )
+
+        logits = model(
+           dense_features=features,
+           sparse_features=sparse_features,
+        )
+    """
+
+    def __init__(
+        self,
+        embedding_bag_collection: EmbeddingBagCollection,
+        dense_in_features: int,
+        num_total_sparse_features:int,
+        dense_arch_layer_sizes: List[int],
+        over_arch_layer_sizes: List[int],
+        dcn_num_layers: int,
+        dcn_low_rank_dim: int,
+        n_emb : int,
+        shard_matrix,
+        shard_strategy,
+        args: argparse.Namespace,
+        dense_device: Optional[torch.device] = None,
+    ) -> None:
+        # initialize DLRM
+        # sparse arch and dense arch are initialized via DLRM
+        super().__init__(
+            embedding_bag_collection,
+            dense_in_features,
+            dense_arch_layer_sizes,
+            over_arch_layer_sizes,
+            dense_device,
+        )
+        
+        self.args = args
+        if ext_dist.my_size > 1:
+            if n_emb < ext_dist.my_size:
+                sys.exit(
+                    "only (%d) sparse features for (%d) devices, table partitions will fail"
+                    % (n_emb, ext_dist.my_size)
+                )
+            self.n_global_emb = n_emb
+            #self.n_local_emb, self.n_emb_per_rank = ext_dist.get_split_lengths(
+            #    n_emb
+            #)
+            self.shard_matrix = shard_matrix
+            self.shard_strategy = shard_strategy
+            self.n_emb_per_rank = [len(s_m) for s_m in shard_matrix]
+            #self.local_emb_slice = ext_dist.get_my_slice(n_emb)
+            #self.local_emb_indices = list(range(n_emb))[self.local_emb_slice]
+
+
+        self.embedding_dim: int = embedding_bag_collection.embedding_bag_configs()[
+            0
+        ].embedding_dim
+        #num_sparse_features: int = len(self.sparse_arch.sparse_feature_names)
+
+        # Fix interaction and over arch for DLRM_DCN
+
+        crossnet = LowRankCrossNet(
+            in_features=(num_total_sparse_features + 1) * self.embedding_dim,
+            num_layers=dcn_num_layers,
+            low_rank=dcn_low_rank_dim,
+        )
+
+        self.inter_arch = InteractionDCNArch(
+            num_sparse_features=num_total_sparse_features,
+            crossnet=crossnet,
+        )
+        self.inter_arch.to(dense_device)
+
+        over_in_features: int = (num_total_sparse_features + 1) * self.embedding_dim
+
+        self.over_arch = OverArch(
+            in_features=over_in_features,
+            layer_sizes=over_arch_layer_sizes,
+            device=dense_device,
+        )
+
+    def sequential_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = self.sparse_arch(sparse_features)
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def distributed_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_sparse = self.sparse_arch(sparse_features)
+        a2a_req = ext_dist.alltoall(embedded_sparse, self.n_emb_per_rank)
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = a2a_req.wait()
+        #['0', '8', '16', '24'], ['1', '9', '17', '25'], ['2', '10', '18'], ['3', '11', '19'], ['4', '12', '20'], ['5', '13', '21'], ['6', '14', '22'], ['7', '15', '23']
+        split_embed = {}
+        for index_rank in range(len(self.shard_matrix)):
+            embed_rank = torch.split(embedded_sparse[index_rank], self.embedding_dim, dim=1)
+            for index in range(len(self.shard_matrix[index_rank])):
+                split_embed[self.shard_matrix[index_rank][index]] = embed_rank[index]
+
+        embedded_sparse = []
+        for i in range(len(self.args.num_embeddings_per_feature)):
+            embedded_sparse.append(split_embed[i])
+        embedded_sparse = torch.cat(list(embedded_sparse), dim=1).reshape(embedded_dense.size(0), -1, embedded_dense.size(-1))
+        if self.args.fp16:
+            embedded_sparse = embedded_sparse.half() if embedded_sparse.dtype != torch.float16 else embedded_sparse
+            embedded_dense = embedded_dense.half() if embedded_dense.dtype != torch.float16 else embedded_dense
+        elif self.args.bf16:
+            embedded_sparse = embedded_sparse.bfloat16() if embedded_sparse.dtype != torch.bfloat16 else embedded_sparse
+            embedded_dense = embedded_dense.bfloat16() if embedded_dense.dtype != torch.bfloat16 else embedded_dense
+
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        if ext_dist.my_size > 1:
+            # multi-node multi-device run
+            return self.distributed_forward(dense_features, sparse_features)
+        else:
+            # single device run
+            return self.sequential_forward(dense_features, sparse_features)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py
new file mode 100644
index 000000000..29ad49965
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py
@@ -0,0 +1,1057 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import itertools
+import os
+import sys
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import List, Optional
+import numpy as np
+
+import torch
+import torchmetrics as metrics
+from torch.autograd.profiler import record_function
+from pyre_extensions import none_throws
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec import EmbeddingBagCollection
+from torchrec.datasets.criteo import DEFAULT_CAT_NAMES, DEFAULT_INT_NAMES
+#from torchrec.distributed import TrainPipelineSparseDist
+from torchrec.distributed.comm import get_local_size
+#from torchrec.distributed.model_parallel import (
+#    DistributedModelParallel,
+#    get_default_sharders,
+#)
+#from torchrec.distributed.planner import EmbeddingShardingPlanner, Topology
+#from torchrec.distributed.planner.storage_reservations import (
+#    HeuristicalStorageReservation,
+#)
+from dist_models import DIST_DLRM_DCN
+from torchrec.models.dlrm import DLRM, DLRM_DCN, DLRM_Projection, DLRMTrain
+from torchrec.modules.embedding_configs import EmbeddingBagConfig
+from torchrec.optim.apply_optimizer_in_backward import apply_optimizer_in_backward
+from torchrec.optim.keyed import CombinedOptimizer, KeyedOptimizerWrapper
+from torchrec.optim.optimizers import in_backward_optimizer_filter
+from tqdm import tqdm
+import extend_distributed as ext_dist
+from extend_distributed import dist, env2int
+from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
+from multi_hot import Multihot, RestartableMap
+from lr_scheduler import LRPolicyScheduler
+from data.dlrm_dataloader import get_dataloader
+import logging
+import sharding
+
+logger: logging.Logger = logging.getLogger(__name__)
+logger.setLevel(1)
+
+# # OSS import
+# try:
+#     # pyre-ignore[21]
+#     # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:dlrm_dataloader
+#     from data.dlrm_dataloader import get_dataloader
+
+#     # pyre-ignore[21]
+#     # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:lr_scheduler
+#     from lr_scheduler import LRPolicyScheduler
+
+#     # pyre-ignore[21]
+#     # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+#     from multi_hot import Multihot, RestartableMap
+# except ImportError:
+#     pass
+
+# # internal import
+# try:
+#     from .data.dlrm_dataloader import get_dataloader  # noqa F811
+#     from .lr_scheduler import LRPolicyScheduler  # noqa F811
+#     from .multi_hot import Multihot, RestartableMap  # noqa F811
+# except ImportError:
+#     pass
+
+TRAIN_PIPELINE_STAGES = 3  # Number of stages in TrainPipelineSparseDist.
+
+n_emb = None
+#local_emb_slice = None
+local_emb_indices = None
+default_embedding_names = None
+idx_name = None
+
+
+class InteractionType(Enum):
+    ORIGINAL = "original"
+    DCN = "dcn"
+    PROJECTION = "projection"
+
+    def __str__(self):
+        return self.value
+
+
+def load_snapshot(model, snapshot_dir):
+    from torchsnapshot import Snapshot
+    snapshot = Snapshot(path=snapshot_dir)
+    snapshot.restore(app_state={"model": model})
+
+
+def parse_args(argv: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
+    parser.add_argument(
+        "--epochs",
+        type=int,
+        default=1,
+        help="number of epochs to train",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=32,
+        help="batch size to use for training",
+    )
+    parser.add_argument(
+        "--drop_last_training_batch",
+        dest="drop_last_training_batch",
+        action="store_true",
+        help="Drop the last non-full training batch",
+    )
+    parser.add_argument(
+        "--test_batch_size",
+        type=int,
+        default=None,
+        help="batch size to use for validation and testing",
+    )
+    parser.add_argument(
+        "--limit_train_batches",
+        type=int,
+        default=None,
+        help="number of train batches",
+    )
+    parser.add_argument(
+        "--limit_val_batches",
+        type=int,
+        default=None,
+        help="number of validation batches",
+    )
+    parser.add_argument(
+        "--limit_test_batches",
+        type=int,
+        default=None,
+        help="number of test batches",
+    )
+    parser.add_argument(
+        "--dataset_name",
+        type=str,
+        choices=["criteo_1t", "criteo_kaggle"],
+        default="criteo_1t",
+        help="dataset for experiment, current support criteo_1tb, criteo_kaggle",
+    )
+    parser.add_argument(
+        "--num_embeddings",
+        type=int,
+        default=100_000,
+        help="max_ind_size. The number of embeddings in each embedding table. Defaults"
+        " to 100_000 if num_embeddings_per_feature is not supplied.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        default=None,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--dense_arch_layer_sizes",
+        type=str,
+        default="512,256,64",
+        help="Comma separated layer sizes for dense arch.",
+    )
+    parser.add_argument(
+        "--over_arch_layer_sizes",
+        type=str,
+        default="512,512,256,1",
+        help="Comma separated layer sizes for over arch.",
+    )
+    parser.add_argument(
+        "--embedding_dim",
+        type=int,
+        default=64,
+        help="Size of each embedding.",
+    )
+    parser.add_argument(
+        "--interaction_branch1_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch1 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--interaction_branch2_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch2 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--dcn_num_layers",
+        type=int,
+        default=3,
+        help="Number of DCN layers in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--dcn_low_rank_dim",
+        type=int,
+        default=512,
+        help="Low rank dimension for DCN in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--undersampling_rate",
+        type=float,
+        help="Desired proportion of zero-labeled samples to retain (i.e. undersampling zero-labeled rows)."
+        " Ex. 0.3 indicates only 30pct of the rows with label 0 will be kept."
+        " All rows with label 1 will be kept. Value should be between 0 and 1."
+        " When not supplied, no undersampling occurs.",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        help="Random seed for reproducibility.",
+    )
+    parser.add_argument(
+        "--pin_memory",
+        dest="pin_memory",
+        action="store_true",
+        help="Use pinned memory when loading data.",
+    )
+    parser.add_argument(
+        "--mmap_mode",
+        dest="mmap_mode",
+        action="store_true",
+        help="--mmap_mode mmaps the dataset."
+        " That is, the dataset is kept on disk but is accessed as if it were in memory."
+        " --mmap_mode is intended mostly for faster debugging. Use --mmap_mode to bypass"
+        " preloading the dataset when preloading takes too long or when there is "
+        " insufficient memory available to load the full dataset.",
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the Criteo dataset npy files.",
+    )
+    parser.add_argument(
+        "--synthetic_multi_hot_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the MLPerf v2 synthetic multi-hot dataset npz files.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=15.0,
+        help="Learning rate.",
+    )
+    parser.add_argument(
+        "--eps",
+        type=float,
+        default=1e-8,
+        help="Epsilon for Adagrad optimizer.",
+    )
+    parser.add_argument(
+        "--shuffle_batches",
+        dest="shuffle_batches",
+        action="store_true",
+        help="Shuffle each batch during training.",
+    )
+    parser.add_argument(
+        "--shuffle_training_set",
+        dest="shuffle_training_set",
+        action="store_true",
+        help="Shuffle the training set in memory. This will override mmap_mode",
+    )
+    parser.add_argument(
+        "--validation_freq_within_epoch",
+        type=int,
+        default=None,
+        help="Frequency at which validation will be run within an epoch.",
+    )
+    parser.set_defaults(
+        pin_memory=None,
+        mmap_mode=None,
+        drop_last=None,
+        shuffle_batches=None,
+        shuffle_training_set=None,
+    )
+    parser.add_argument(
+        "--adagrad",
+        dest="adagrad",
+        action="store_true",
+        help="Flag to determine if adagrad optimizer should be used.",
+    )
+    parser.add_argument(
+        "--interaction_type",
+        type=InteractionType,
+        choices=list(InteractionType),
+        default=InteractionType.ORIGINAL,
+        help="Determine the interaction type to be used (original, dcn, or projection)"
+        " default is original DLRM with pairwise dot product",
+    )
+    parser.add_argument(
+        "--collect_multi_hot_freqs_stats",
+        dest="collect_multi_hot_freqs_stats",
+        action="store_true",
+        help="Flag to determine whether to collect stats on freq of embedding access.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        default=None,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default=None,
+        help="Multi-hot distribution options.",
+    )
+    parser.add_argument("--lr_warmup_steps", type=int, default=0)
+    parser.add_argument("--lr_decay_start", type=int, default=0)
+    parser.add_argument("--lr_decay_steps", type=int, default=0)
+    parser.add_argument(
+        "--print_lr",
+        action="store_true",
+        help="Print learning rate every iteration.",
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help="Enable TensorFloat-32 mode for matrix multiplications on A100 (or newer) GPUs.",
+    )
+    parser.add_argument(
+        "--print_sharding_plan",
+        action="store_true",
+        help="Print the sharding plan used for each embedding table.",
+    )
+    parser.add_argument("--use_xpu", action="store_true", default=False)
+    parser.add_argument(
+        "--log_freq",
+        type=int,
+        default=0,
+        help="log_freq to print performance statistics.",
+    )
+    parser.add_argument("--numpy_rand_seed", type=int, default=123)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--fp16", action="store_true", default=False)
+    parser.add_argument("--amp", action="store_true", default=False)
+    parser.add_argument("--inference_only", action="store_true", default=False)
+    parser.add_argument("--train_with_val", action="store_true", default=False)
+    parser.add_argument("--num_batches", type=int, default=0)
+    parser.add_argument("--num_nodes", type=int, default=1)
+    parser.add_argument("--trans_data", action="store_true", default=False)
+    parser.add_argument(
+    "--sharding_plan",
+    help="Sharding plan to use",
+    type=str,
+    choices=["round_robin", "uniform", "auto", "hier_auto", "custom", "max_min"],
+    default="round_robin",)
+    parser.add_argument(
+        "--snapshot_dir",
+        help="Weight path for dlrm-v2",
+        type=str,
+    )
+
+
+    return parser.parse_args(argv)
+
+
+def _evaluate(
+    limit_batches: Optional[int],
+    #eval_pipeline: TrainPipelineSparseDist,
+    eval_model,
+    eval_dataloader: DataLoader,
+    stage: str,
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> float:
+    """
+    Evaluates model. Computes and prints AUROC. Helper function for train_val_test.
+
+    Args:
+        limit_batches (Optional[int]): Limits the dataloader to the first `limit_batches` batches.
+        eval_pipeline (TrainPipelineSparseDist): pipelined model.
+        eval_dataloader (DataLoader): Dataloader for either the validation set or test set.
+        stage (str): "val" or "test".
+
+    Returns:
+        float: auroc result
+    """
+    eval_model.eval()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+
+    def eval_step(model, batch):
+        batch_size = batch.dense_features.size(0)
+        if args.amp:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        batch.labels = batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+        for k, v in batch.sparse_features.items():
+            batch.sparse_features[k]['values'] = batch.sparse_features[k]['values'].to(device)
+            batch.sparse_features[k]['offsets'] = batch.sparse_features[k]['offsets'].to(device)
+
+        if args.use_device == "xpu":
+            with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                logits = model(batch.dense_features, batch.sparse_features)
+        elif args.use_device == "cuda":
+            with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                logits = model(batch.dense_features, batch.sparse_features)
+        else:
+            pass
+        return logits, batch.labels
+    # Set eval_pipeline._connected to False to cause the pipeline to refill with new batches as if it were newly created and empty.
+    #eval_pipeline._connected = False
+
+    iterator = itertools.islice(iter(eval_dataloader), limit_batches)
+
+    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+
+    is_rank_zero = dist.get_rank() == 0
+    #if is_rank_zero:
+    #    pbar = tqdm(
+    #        iter(int, 1),
+    #        desc=f"Evaluating {stage} set",
+    #        total=len(eval_dataloader),
+    #        disable=False,
+    #    )
+    total_t = 0
+    it = 1
+    with torch.no_grad():
+        while True:
+            try:
+                batch = next(iterator)
+                batch.sparse_features = unpack(batch.sparse_features, default_embedding_names)
+                ext_dist.barrier()
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+                t1 = time.time()
+                with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                    logits, labels = eval_step(eval_model, batch)
+                    #_loss, logits, labels = eval_pipeline.progress(iterator)
+                    preds = torch.sigmoid(logits)
+                    auroc(preds, labels)
+                    if args.use_device == "xpu":
+                        torch.xpu.synchronize()
+                    elif args.use_device == "cuda":
+                        torch.cuda.synchronize()
+                    else:
+                        pass
+                    if it > 5 and args.inference_only:
+                        total_t += time.time() - t1
+                        #if log_freq != 0 and it % log_freq == 0:
+                        logger.info(f"avg eval time per iter at ITER: {it - 5}, {total_t/(it-5)} s")
+            except StopIteration:
+                break
+            if args.enable_profiling and it == 35 and args.inference_only:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_inference_' + rank_index + 'profiling'
+                if args.use_xpu:
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            #if is_rank_zero:
+            #    pbar.update(1)
+            it += 1
+            if it > args.num_batches:
+                break;
+
+    auroc_result = auroc.compute().item()
+    num_samples = torch.tensor(sum(map(len, auroc.target)), device=device)
+    dist.reduce(num_samples, 0, op=dist.ReduceOp.SUM)
+
+    if is_rank_zero:
+        print(f"AUROC over {stage} set: {auroc_result}.")
+        print(f"Number of {stage} samples: {num_samples}")
+    return auroc_result
+
+def _train(
+    train_model,
+    train_optimizer,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    epoch: int,
+    lr_scheduler,
+    print_lr: bool,
+    validation_freq: Optional[int],
+    limit_train_batches: Optional[int],
+    limit_val_batches: Optional[int],
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> bool:
+    """
+    Trains model for 1 epoch. Helper function for train_val_test.
+    Args:
+        train_pipeline (TrainPipelineSparseDist): pipelined model used for training.
+        val_pipeline (TrainPipelineSparseDist): pipelined model used for validation.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        epoch (int): The number of complete passes through the training set so far.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+        print_lr (bool): Whether to print the learning rate every training step.
+        validation_freq (Optional[int]): The number of training steps between validation runs within an epoch.
+        limit_train_batches (Optional[int]): Limits the training set to the first `limit_train_batches` batches.
+        limit_val_batches (Optional[int]): Limits the validation set to the first `limit_val_batches` batches.
+        print_progress (bool): Whether to print tqdm progress bar.
+    Returns:
+        bool: Whether the validation_auroc threshold is reached.
+    """
+    train_model.train()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    def train_step(model, opt, next_batch):
+        batch_size = next_batch.dense_features.size(0)
+        opt.zero_grad(set_to_none=True)
+        if args.amp:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        next_batch.labels = next_batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+
+        for k, v in next_batch.sparse_features.items():
+            next_batch.sparse_features[k]['values'] = next_batch.sparse_features[k]['values'].to(device)
+            next_batch.sparse_features[k]['offsets'] = next_batch.sparse_features[k]['offsets'].to(device)
+        if args.use_device == "xpu":
+            with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                losses, _ = model(next_batch)
+                loss = torch.sum(losses, dim=0)
+        elif args.use_device == "cuda":
+            with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                losses, _ = model(next_batch)
+                loss = torch.sum(losses, dim=0)
+        else:
+            pass
+        loss.backward()
+        opt.step()
+    iterator = itertools.islice(iter(train_dataloader), limit_train_batches)
+
+    is_rank_zero = dist.get_rank() == 0
+    #if is_rank_zero:
+    #    pbar = tqdm(
+    #        iter(int, 1),
+    #        desc=f"Epoch {epoch}",
+    #        total=len(train_dataloader),
+    #        disable=False,
+    #    )
+
+    total_t = 0
+    for it in itertools.count(1):
+        try:
+            if is_rank_zero and print_lr:
+                for i, g in enumerate(train_pipeline._optimizer.param_groups):
+                    print(f"lr: {it} {i} {g['lr']:.6f}")
+            next_batch = next(iterator)
+            next_batch.sparse_features = unpack(next_batch.sparse_features, default_embedding_names)
+            ext_dist.barrier()
+            if args.use_device == "xpu":
+                torch.xpu.synchronize()
+            elif args.use_device == "cuda":
+                torch.cuda.synchronize()
+            else:
+                pass
+            t1 = time.time()
+            with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True, record_shapes=False) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                train_step(train_model, train_optimizer, next_batch)
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+
+                if it > 5:
+                    total_t += time.time() - t1
+                    #if log_freq != 0 and it % log_freq == 0:
+                    logger.info(f"avg training time per iter at ITER: {it - 5}, {total_t/(it - 5)} s")
+
+            if args.enable_profiling and it == 35:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_training_' + rank_index + 'profiling'
+                if args.use_device == "xpu":
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            lr_scheduler.step()
+            #if is_rank_zero:
+            #    pbar.update(1)
+            if validation_freq and (it % validation_freq == 0) and args.train_with_val:
+                auroc_result = _evaluate(
+                    limit_val_batches,
+                    train_model.model,
+                    val_dataloader,
+                    "val",
+                    log_freq, 
+                    device, 
+                    args
+                )
+                if is_rank_zero:
+                    logger.info("The val auc = {auroc_result}")
+                train_model.train()
+        except StopIteration:
+            # Dataset traversal complete
+            break
+        it += 1
+        if it > args.num_batches:
+            break;
+
+    logger.info(f"Total number of iterations: {it - 1}")
+
+
+@dataclass
+class TrainValTestResults:
+    val_aurocs: List[float] = field(default_factory=list)
+    test_auroc: Optional[float] = None
+
+
+def train_val_test(
+    args: argparse.Namespace,
+    model: torch.nn.Module,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    test_dataloader: DataLoader,
+    lr_scheduler: LRPolicyScheduler,
+) -> TrainValTestResults:
+    """
+    Train/validation/test loop.
+
+    Args:
+        args (argparse.Namespace): parsed command line args.
+        model (torch.nn.Module): model to train.
+        optimizer (torch.optim.Optimizer): optimizer to use.
+        device (torch.device): device to use.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        test_dataloader (DataLoader): Test set's dataloader.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+
+    Returns:
+        TrainValTestResults.
+    """
+    results = TrainValTestResults()
+    #train_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    #val_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    #test_pipeline = TrainPipelineSparseDist(model, optimizer, device)
+    if args.fp16 and (not args.amp):
+        model.to(torch.float16)
+    if args.bf16 and (not args.amp):
+        model.to(torch.bfloat16)
+
+    if not args.inference_only:
+        for epoch in range(args.epochs):
+            _train(
+                model,
+                optimizer,
+                train_dataloader,
+                val_dataloader,
+                epoch,
+                lr_scheduler,
+                args.print_lr,
+                args.validation_freq_within_epoch,
+                args.limit_train_batches,
+                args.limit_val_batches,
+                args.log_freq,
+                device,
+                args
+            )
+            if args.train_with_val:
+                val_auroc = _evaluate(
+                    args.limit_val_batches, model.model, val_dataloader, "val", args.log_freq, device, args
+                )
+                results.val_aurocs.append(val_auroc)
+
+    else:
+        test_auroc = _evaluate(
+            args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+        )
+        results.test_auroc = test_auroc
+
+    return results
+
+
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+    args = parse_args(argv)
+    args.amp_dtype = torch.float32
+    if args.bf16:
+        args.amp_dtype = torch.bfloat16
+    if args.fp16:
+        args.amp_dtype = torch.float16
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    args.enable_profiling = False
+    if os.getenv('PROFILE') == '1': args.enable_profiling=True
+
+    if args.use_xpu:
+        import intel_extension_for_pytorch
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError as e:
+            pass
+
+    if not args.use_xpu:
+        torch.backends.cuda.matmul.allow_tf32 = args.allow_tf32
+
+    if args.multi_hot_sizes is not None and args.trans_data:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or (not args.trans_data) or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    np.random.seed(args.numpy_rand_seed)
+    torch.manual_seed(args.numpy_rand_seed)
+
+    #rank = int(os.environ["LOCAL_RANK"])
+    rank = env2int(
+        [
+            "MPI_LOCALRANKID",
+            "OMPI_COMM_WORLD_LOCAL_RANK",
+            "MV2_COMM_WORLD_LOCAL_RANK",
+            "LOCAL_RANK",
+        ],
+        0,
+    )
+    args.use_device = "cpu"
+    if args.use_xpu and torch.xpu.is_available():
+        torch.xpu.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device("xpu", rank)
+        backend = "ccl"
+        torch.xpu.set_device(device)
+        args.use_device = "xpu"
+    elif torch.cuda.is_available():
+        torch.cuda.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device(f"cuda:{rank}")
+        backend = "nccl"
+        torch.cuda.set_device(device)
+        args.use_device = "cuda"
+    else:
+        device: torch.device = torch.device("cpu")
+        backend = "gloo"
+
+    if rank == 0:
+        print(
+            "PARAMS: (lr, batch_size, warmup_steps, decay_start, decay_steps): "
+            f"{(args.learning_rate, args.batch_size, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps)}"
+        )
+    ext_dist.init_distributed(
+        local_rank=-1, use_device=args.use_device, backend=backend
+    )
+
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path
+        is args.synthetic_multi_hot_criteo_path
+        is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    train_dataloader = get_dataloader(args, backend, "train")
+    val_dataloader = get_dataloader(args, backend, "val")
+    test_dataloader = get_dataloader(args, backend, "test")
+    # use dist_models, if not you can comment out
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    n_emb = len(DEFAULT_CAT_NAMES)
+    #if dist.get_world_size() == 8 and dist.get_rank() == 6:
+    #    local_emb_indices[1] = 25
+    #if dist.get_world_size() == 8 and dist.get_rank() == 7:
+    #    local_emb_indices[2] = 21
+
+    shard_matrix_str, shard_strategy_str, shard_matrix, shard_strategy = sharding.generate_plan(
+    args.num_embeddings_per_feature, args.multi_hot_sizes, args.num_nodes, dist.get_world_size(), args, dist.get_rank() == 0)
+    #local_emb_slice = ext_dist.get_my_slice(n_emb)
+    local_emb_indices = shard_matrix[rank]
+    default_embedding_names = [DEFAULT_CAT_NAMES[i] for i in local_emb_indices]
+    idx_name = zip(local_emb_indices, default_embedding_names)
+
+    eb_configs = [
+        EmbeddingBagConfig(
+            name=f"t_{feature_name}",
+            embedding_dim=args.embedding_dim,
+            num_embeddings=none_throws(args.num_embeddings_per_feature)[feature_idx]
+            if args.num_embeddings is None
+            else args.num_embeddings,
+            feature_names=[feature_name],
+        )
+        #for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
+        for feature_idx, feature_name in idx_name
+    ]
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    if args.interaction_type == InteractionType.ORIGINAL:
+        dlrm_model = DLRM(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.DCN:
+        #dlrm_model = DLRM_DCN(
+        dlrm_model = DIST_DLRM_DCN(
+            embedding_bag_collection=EmbeddingBagCollection(
+                #tables=eb_configs, device=torch.device("meta")
+                tables=eb_configs, device="cpu"
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            num_total_sparse_features=n_emb,
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dcn_num_layers=args.dcn_num_layers,
+            dcn_low_rank_dim=args.dcn_low_rank_dim,
+            n_emb=n_emb,
+            shard_matrix=shard_matrix, 
+            shard_strategy = shard_strategy,
+            args=args,
+            dense_device="cpu",
+        )
+    elif args.interaction_type == InteractionType.PROJECTION:
+        dlrm_model = DLRM_Projection(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
+            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
+            dense_device=device,
+        )
+    else:
+        raise ValueError(
+            "Unknown interaction option set. Should be original, dcn, or projection."
+        )
+
+    if not args.amp:
+        dlrm_model.to(args.amp_dtype)
+    train_model = DLRMTrain(dlrm_model)
+    if args.inference_only:
+        assert args.snapshot_dir
+        load_snapshot(train_model, args.snapshot_dir)
+
+    #embedding_optimizer = torch.optim.Adagrad if args.adagrad else torch.optim.SGD
+    # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
+    # the optimizer update will be applied in the backward pass, in this case through a fused op.
+    # TorchRec will use the FBGEMM implementation of EXACT_ADAGRAD. For GPU devices, a fused CUDA kernel is invoked. For CPU, FBGEMM_GPU invokes CPU kernels
+    # https://github.com/pytorch/FBGEMM/blob/2cb8b0dff3e67f9a009c4299defbd6b99cc12b8f/fbgemm_gpu/fbgemm_gpu/split_table_batched_embeddings_ops.py#L676-L678
+
+    # Note that lr_decay, weight_decay and initial_accumulator_value for Adagrad optimizer in FBGEMM v0.3.2
+    # cannot be specified below. This equivalently means that all these parameters are hardcoded to zero.
+    optimizer_kwargs = {"lr": args.learning_rate}
+    if args.adagrad:
+        optimizer_kwargs["eps"] = args.eps
+    #apply_optimizer_in_backward(
+    #    embedding_optimizer,
+    #    train_model.model.sparse_arch.parameters(),
+    #    optimizer_kwargs,
+    #)
+    #planner = EmbeddingShardingPlanner(
+    #    topology=Topology(
+    #        local_world_size=get_local_size(),
+    #        world_size=dist.get_world_size(),
+    #        compute_device=device.type,
+    #    ),
+    #    batch_size=args.batch_size,
+    #    # If experience OOM, increase the percentage. see
+    #    # https://pytorch.org/torchrec/torchrec.distributed.planner.html#torchrec.distributed.planner.storage_reservations.HeuristicalStorageReservation
+    #    storage_reservation=HeuristicalStorageReservation(percentage=0.05),
+    #)
+    #plan = planner.collective_plan(
+    #    train_model, get_default_sharders(), dist.GroupMember.WORLD
+    #)
+
+    #model = DistributedModelParallel(
+    #    module=train_model,
+    #    device=device,
+    #    plan=plan,
+    #)
+    #if rank == 0 and args.print_sharding_plan:
+    #    for collectionkey, plans in model._plan.plan.items():
+    #        print(collectionkey)
+    #        for table_name, plan in plans.items():
+    #            print(table_name, "\n", plan, "\n")
+
+    model = train_model
+    model.model.sparse_arch = SparseArchTraceAbleWrapper(model.model.sparse_arch)
+
+
+    if ext_dist.my_size > 1:
+        if args.use_device == "cuda":
+            device_ids = [rank]
+            #print(device_ids)
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+        elif args.use_device == "xpu":
+            #device_ids = ["xpu:{}".format(ext_dist.my_local_rank)]
+            device_ids = [device]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+        else:
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch)
+
+    print(model)
+    #def optimizer_with_params():
+    #    if args.adagrad:
+    #        return lambda params: torch.optim.Adagrad(params, lr=args.learning_rate, eps=args.eps)
+    #    else:
+    #        return lambda params: torch.optim.SGD(params, lr=args.learning_rate)
+
+    #optimizer = KeyedOptimizerWrapper(
+    #    dict(model.named_parameters()),
+    #    optimizer_with_params(),
+    #)
+    #for k, v in optimizer.state_dict()['state'].items():
+    #    print(k)
+    #    print(v['step'].device)
+    #    print(v['sum'].device)
+    #exit()
+    #dense_optimizer = KeyedOptimizerWrapper(
+    #    dict(in_backward_optimizer_filter(model.named_parameters())),
+    #    optimizer_with_params(),
+    #)
+    #optimizer = CombinedOptimizer([model.fused_optimizer, dense_optimizer])
+    #optimizer = CombinedOptimizer([optimizer])
+    optimizer = None
+    lr_scheduler = None
+    if not args.inference_only:
+        if args.adagrad:
+            optimizer = torch.optim.Adagrad(model.parameters(), lr=args.learning_rate, eps=args.eps)
+        else:
+            optimizer = torch.optim.SGD(model.parameters(), lr=args.learning_rate)
+        lr_scheduler = LRPolicyScheduler(
+            optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
+        )
+
+    if args.use_device == "xpu":
+        if not args.inference_only:
+            model.train()
+            model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                      dtype=torch.float32)
+        else:
+            model.eval()
+            model = torch.xpu.optimize(model=model, dtype=args.amp_dtype)
+
+    if args.trans_data and (args.multi_hot_sizes is not None):
+        multihot = Multihot(
+            args.multi_hot_sizes,
+            args.num_embeddings_per_feature,
+            args.batch_size,
+            collect_freqs_stats=args.collect_multi_hot_freqs_stats,
+            dist_type=args.multi_hot_distribution_type,
+        )
+        multihot.pause_stats_collection_during_val_and_test(model)
+        train_dataloader = RestartableMap(
+            multihot.convert_to_multi_hot, train_dataloader
+        )
+        val_dataloader = RestartableMap(multihot.convert_to_multi_hot, val_dataloader)
+        test_dataloader = RestartableMap(multihot.convert_to_multi_hot, test_dataloader)
+    train_val_test(
+        args,
+        model,
+        optimizer,
+        device,
+        train_dataloader,
+        val_dataloader,
+        test_dataloader,
+        lr_scheduler,
+    )
+    if args.collect_multi_hot_freqs_stats:
+        multihot.save_freqs_stats()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py
new file mode 100644
index 000000000..865eba420
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py
@@ -0,0 +1,663 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import builtins
+import os
+import sys
+
+import torch
+import torch.distributed as dist
+from torch.autograd import Function
+from torch.autograd.profiler import record_function
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+try:
+    import intel_extension_for_pytorch
+except ImportError as e:
+    pass
+
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+        torch_ccl = False
+    else:
+        import torch_ccl
+except ImportError as e:
+    # print(e)
+    torch_ccl = False
+
+try:
+    import torch_ucc
+except ImportError as e:
+    torch_ucc = False
+
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+a2a_impl = os.environ.get("DLRM_ALLTOALL_IMPL", "")
+
+myreq = None
+
+
+def env2int(env_list, default=-1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0:
+            return val
+    return default
+
+
+def get_my_slice(n):
+    tmp_size = dist.get_world_size()
+    tmp_rank = dist.get_rank()
+
+    k, m = divmod(n, tmp_size)
+    return slice(
+        tmp_rank * k + min(tmp_rank, m), (tmp_rank + 1) * k + min(tmp_rank + 1, m), 1
+    )
+
+
+def get_split_lengths(n):
+    if my_size >1:
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+    else:
+        tmp_size = my_size
+        tmp_rank = my_rank
+
+    k, m = divmod(n, tmp_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        splits = [(k + 1) if i < m else k for i in range(tmp_size)]
+        my_len = splits[tmp_rank]
+    return (my_len, splits)
+
+
+def init_distributed(rank=-1, local_rank=-1, size=-1, use_device="cpu", backend=""):
+    global myreq
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(
+        ["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_SIZE", "WORLD_SIZE"]
+    )
+    if backend == "" and num_mpi_ranks > 1:
+        if torch_ccl or oneccl_bindings_for_pytorch:
+            backend = "ccl"
+        elif use_device=="cuda" and dist.is_nccl_available():
+            backend = "nccl"
+        elif dist.is_mpi_available():
+            backend = "mpi"
+        else:
+            print(
+                "WARNING: MPI multi-process launch detected but PyTorch MPI backend not available."
+            )
+            backend = "gloo"
+
+    if backend != "":
+        # guess Rank and size
+        if rank == -1:
+            rank = env2int(
+                ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+            )
+        if size == -1:
+            size = env2int(
+                [
+                    "PMI_SIZE",
+                    "OMPI_COMM_WORLD_SIZE",
+                    "MV2_COMM_WORLD_SIZE",
+                    "WORLD_SIZE",
+                ],
+                1,
+            )
+        if not os.environ.get("RANK", None) and rank != -1:
+            os.environ["RANK"] = str(rank)
+        if not os.environ.get("WORLD_SIZE", None) and size != -1:
+            os.environ["WORLD_SIZE"] = str(size)
+        if not os.environ.get("MASTER_PORT", None):
+            os.environ["MASTER_PORT"] = "29500"
+        if not os.environ.get("MASTER_ADDR", None):
+            local_size = env2int(
+                [
+                    "MPI_LOCALNRANKS",
+                    "OMPI_COMM_WORLD_LOCAL_SIZE",
+                    "MV2_COMM_WORLD_LOCAL_SIZE",
+                ],
+                1,
+            )
+            if local_size != size and backend != "mpi":
+                print(
+                    "Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default"
+                )
+                print(
+                    "If this run hangs, try exporting rank 0's hostname as MASTER_ADDR"
+                )
+            os.environ["MASTER_ADDR"] = "127.0.0.1"
+
+    if size > 1:
+        if local_rank == -1:
+            my_local_rank = env2int(
+                [
+                    "MPI_LOCALRANKID",
+                    "OMPI_COMM_WORLD_LOCAL_RANK",
+                    "MV2_COMM_WORLD_LOCAL_RANK",
+                    "LOCAL_RANK",
+                ],
+                0,
+            )
+        else:
+            my_local_rank = local_rank
+        my_local_size = env2int(
+            [
+                "MPI_LOCALNRANKS",
+                "OMPI_COMM_WORLD_LOCAL_SIZE",
+                "MV2_COMM_WORLD_LOCAL_SIZE",
+            ],
+            1,
+        )
+        if use_device=="cuda":
+            if my_local_size > torch.cuda.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.cuda.device_count())
+                )
+                sys.exit(1)
+            torch.cuda.set_device(my_local_rank)
+        elif use_device=="xpu":
+            if my_local_size > torch.xpu.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.xpu.device_count())
+                )
+                sys.exit(1)
+            torch.xpu.set_device(my_local_rank)
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        this_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        #my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        #my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if this_rank == 0:
+            print("Running on %d ranks using %s backend" % (my_size, backend))
+        if hasattr(dist, "all_to_all_single"):
+            try:
+                if use_device == "xpu":
+                    test_device: torch.device = torch.device("xpu", dist.get_rank())
+                elif use_device == "cuda":
+                    test_device: torch.device = torch.device(f"cuda:{dist.get_rank()}")
+                else:
+                    pass
+
+                a = torch.rand([1, 3]).to(test_device)
+                dist.broadcast(a, src=0)
+                print("Successfully execute distributed broadcast on the xpu:{}".format(this_rank))
+
+                tensor_list = [torch.zeros(2, dtype=torch.int64).to(test_device) for _ in range(my_size)]
+                tensor = (torch.arange(2, dtype=torch.int64) + 1 + 2 * rank).to(test_device)
+                dist.all_gather(tensor_list, tensor)
+                print("Successfully execute distributed all_gather on the xpu:{}".format(this_rank))
+
+                t = torch.zeros([4*my_size])
+                if use_device == "cuda":
+                    t = t.cuda()
+                elif  use_device == "xpu":
+                    t = t.xpu()
+                dist.all_to_all_single(t, t)
+
+                print("Successfully execute distributed all_to_all_single on the xpu:{}".format(this_rank))
+                alltoall_supported = True
+            except RuntimeError as err:
+                print("fail to enable all_to_all_single primitive: %s" % err)
+                exit()
+        if a2a_impl == "alltoall" and alltoall_supported == False:
+            print(
+                "Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall"
+                % (a2a_impl, backend)
+            )
+            a2a_impl = "scatter"
+        if a2a_impl != "":
+            print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+    else:
+        this_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    print_all(
+        "world size: %d, current rank: %d, local rank: %d"
+        % (my_size, this_rank, my_local_rank)
+    )
+    myreq = Request()
+
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            for j in range(table_split_lengths[i]):
+                out_tensor = inputs[0].new_empty(
+                    [a2a_info.local_batch_num, a2a_info.emb_dim]
+                )
+                scatter_list = (
+                    list(inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else []
+                )
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        a2a_info = ctx.a2a_info
+        grad_output = [t.contiguous() for t in grad_output]
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else [a2a_info.local_batch_num] * tmp_size
+        )
+        per_rank_table_splits = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_inputs = [
+            grad_output[0].new_empty([ctx.a2a_info.batch_size, ctx.a2a_info.emb_dim])
+            for _ in range(a2a_info.local_table_num)
+        ]
+        req_list = []
+        ind = 0
+        for i in range(tmp_size):
+            for j in range(per_rank_table_splits[i]):
+                gather_list = (
+                    list(grad_inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else None
+                )
+                req = dist.gather(grad_output[ind], gather_list, dst=i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(batch_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            out_tensor = input.new_empty(
+                [a2a_info.local_batch_num, table_split_lengths[i] * a2a_info.emb_dim]
+            )
+            req = dist.scatter(
+                out_tensor, scatter_list if i == tmp_rank else [], src=i, async_op=True
+            )
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        ctx.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2a_info.emb_dim, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        assert len(grad_output) == tmp_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2a_info = ctx.a2a_info
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_input = grad_output[0].new_empty(
+            [a2a_info.batch_size, a2a_info.emb_dim * a2a_info.local_table_num]
+        )
+        gather_list = list(grad_input.split(batch_split_lengths, dim=0))
+        req_list = []
+        for i in range(tmp_size):
+            req = dist.gather(
+                scatter_list[i],
+                gather_list if i == tmp_rank else [],
+                dst=i,
+                async_op=True,
+            )
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        with record_function("DLRM alltoall_req_fwd_single"):
+            batch_split_lengths = a2a_info.global_batch_partition_slices
+            if batch_split_lengths:
+                batch_split_lengths = [
+                    m * a2a_info.emb_dim * a2a_info.local_table_num
+                    for m in batch_split_lengths
+                ]
+            table_split_lengths = a2a_info.global_table_wise_parition_slices
+            if table_split_lengths:
+                table_split_lengths = [
+                    a2a_info.local_batch_num * e * a2a_info.emb_dim
+                    for e in table_split_lengths
+                ]
+            input = torch.cat(inputs, dim=1).view([-1])
+            output = input.new_empty(
+                [
+                    a2a_info.global_table_num
+                    * a2a_info.local_batch_num
+                    * a2a_info.emb_dim
+                ]
+            )
+            req = dist.all_to_all_single(
+                output, input, table_split_lengths, batch_split_lengths, async_op=True
+            )
+
+            myreq.req = req
+            myreq.tensor = []
+            myreq.tensor.append(output)
+            myreq.tensor = tuple(myreq.tensor)
+            a2a_info.batch_split_lengths = batch_split_lengths
+            a2a_info.table_split_lengths = table_split_lengths
+            myreq.a2a_info = a2a_info
+            ctx.a2a_info = a2a_info
+            return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        with record_function("DLRM alltoall_req_bwd_single"):
+            a2a_info = ctx.a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            grad_input = myreq.tensor
+            grad_inputs = grad_input.view([a2a_info.batch_size, -1]).split(
+                a2a_info.emb_dim, dim=1
+            )
+            grad_inputs = [gin.contiguous() for gin in grad_inputs]
+            myreq.tensor = None
+            return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        with record_function("DLRM alltoall_wait_fwd_single"):
+            a2a_info = myreq.a2a_info
+            ctx.a2a_info = a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            myreq.tensor = None
+            table_split_lengths = (
+                a2a_info.table_split_lengths
+                if a2a_info.table_split_lengths
+                else a2a_info.local_table_num
+                * a2a_info.local_batch_num
+                * a2a_info.emb_dim
+            )
+            outputs = output[0].split(table_split_lengths)
+            outputs = tuple(
+                [out.view([a2a_info.local_batch_num, -1]) for out in outputs]
+            )
+            return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        with record_function("DLRM alltoall_wait_bwd_single"):
+            a2a_info = ctx.a2a_info
+            grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+            grad_output = torch.cat(grad_outputs)
+            grad_input = grad_output.new_empty(
+                [a2a_info.batch_size * a2a_info.local_table_num * a2a_info.emb_dim]
+            )
+            req = dist.all_to_all_single(
+                grad_input,
+                grad_output,
+                a2a_info.batch_split_lengths,
+                a2a_info.table_split_lengths,
+                async_op=True,
+            )
+            myreq.req = req
+            myreq.tensor = grad_input
+            return (grad_output,)
+
+
+class AllGather(Function):
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * tmp_size
+
+        assert len(global_lengths) == tmp_size
+        assert global_lengths[tmp_rank] == input.size(dim)
+        local_start = sum(global_lengths[:tmp_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[tmp_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(tmp_size)]
+            gather_list = []
+            for length in global_lengths:
+                output_size[dim] = length
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+
+class All2AllInfo(object):
+    pass
+
+
+def alltoall(inputs, per_rank_table_splits):
+    tmp_size = dist.get_world_size()
+    global myreq
+    batch_size, emb_dim = inputs[0].size()
+    a2a_info = All2AllInfo()
+    a2a_info.local_table_num = len(inputs)
+    a2a_info.global_table_wise_parition_slices = per_rank_table_splits
+    (
+        a2a_info.local_batch_num,
+        a2a_info.global_batch_partition_slices,
+    ) = get_split_lengths(batch_size)
+    a2a_info.emb_dim = emb_dim
+    a2a_info.batch_size = batch_size
+    a2a_info.global_table_num = (
+        sum(per_rank_table_splits)
+        if per_rank_table_splits
+        else a2a_info.local_table_num * tmp_size
+    )
+
+    if a2a_impl == "" and alltoall_supported or a2a_impl == "alltoall":
+        # print("Using All2All_Req")
+        output = All2All_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == "" or a2a_impl == "scatter":
+        # print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == "scatter_list":
+        # print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print(
+            "Unknown value set for DLRM_ALLTOALL_IMPL (%s), "
+            "please use one of [alltoall, scatter, scatter_list]" % a2a_impl
+        )
+    return myreq
+
+
+def all_gather(input, lengths, dim=0):
+    tmp_size = dist.get_world_size()
+    if not lengths:
+        lengths = [input.size(0)] * tmp_size
+    return AllGather.apply(input, lengths, dim)
+
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
+
+# Override builtin print function to print only from rank 0
+orig_print = builtins.print
+#
+#
+#def rank0_print(*args, **kwargs):
+#    if my_rank <= 0 or kwargs.get("print_all", False):
+#        orig_print(*args, **kwargs)
+#
+#
+#builtins.print = rank0_print
+#
+# Allow printing from all rank with explicit print_all
+def print_all(*args, **kwargs):
+    orig_print(*args, **kwargs)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py
new file mode 100644
index 000000000..673111dc0
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py
@@ -0,0 +1,67 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+from torchrec.models.dlrm import SparseArch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from typing import List
+
+def unpack(input :KeyedJaggedTensor, default_embedding_names:[str]) -> dict:
+    output = {}
+    for k, v in input.to_dict().items():
+        if k in default_embedding_names:
+            output[k] = {}
+            output[k]['values'] = v._values.to(torch.int64)
+            output[k]['offsets'] = v._offsets.to(torch.int64)
+    return output
+
+
+class SparseArchTraceAbleWrapper(nn.Module):
+
+    def __init__(self, sparse_arch: SparseArch) -> None:
+        super().__init__()
+        self.sparse_arch: SparseArch = sparse_arch
+        for embedding_bag in self.sparse_arch.embedding_bag_collection.embedding_bags.values():
+            embedding_bag.sparse = True
+
+    def forward(
+        self,
+        features: dict,
+    ) -> torch.Tensor:
+        """
+        Args:
+            features (dict): input tensors of sparse features.
+        Returns:
+            torch.Tensor: tensor of shape B X F X D.
+        """
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        pooled_embeddings: List[torch.Tensor] = []
+        for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+            for feature_name in embedding_bag_collection._feature_names[i]:
+                f = features[feature_name]
+                res = embedding_bag(
+                    f['values'],
+                    f['offsets'],
+                    per_sample_weights=None,
+                )
+                pooled_embeddings.append(res)
+        #data = torch.cat(pooled_embeddings, dim=1)
+        #B: int = features['cat_0']['offsets'].numel() - 1
+        #return data.reshape(B, self.sparse_arch.F, self.sparse_arch.D)
+        return pooled_embeddings
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py
new file mode 100644
index 000000000..4aebce416
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Copied from https://github.com/facebookresearch/dlrm/blob/mlperf/dlrm_s_pytorch.py
+
+import sys
+
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_hot.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_hot.py
new file mode 100644
index 000000000..77b13b9e8
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_hot.py
@@ -0,0 +1,175 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import List, Tuple
+
+import numpy as np
+
+import torch
+from torchrec.datasets.utils import Batch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class RestartableMap:
+    def __init__(self, f, source):
+        self.source = source
+        self.func = f
+
+    def __iter__(self):
+        for x in self.source:
+            yield self.func(x)
+
+    def __len__(self):
+        return len(self.source)
+
+
+class Multihot:
+    def __init__(
+        self,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+        collect_freqs_stats: bool,
+        dist_type: str = "uniform",
+    ):
+        if dist_type not in {"uniform", "pareto"}:
+            raise ValueError(
+                "Multi-hot distribution type {} is not supported."
+                'Only "uniform" and "pareto" are supported.'.format(dist_type)
+            )
+        self.dist_type = dist_type
+        self.multi_hot_sizes = multi_hot_sizes
+        self.num_embeddings_per_feature = num_embeddings_per_feature
+        self.batch_size = batch_size
+
+        # Generate 1-hot to multi-hot lookup tables, one lookup table per sparse embedding table.
+        self.multi_hot_tables_l = self.__make_multi_hot_indices_tables(
+            dist_type, multi_hot_sizes, num_embeddings_per_feature
+        )
+
+        # Pooling offsets are computed once and reused.
+        self.offsets = self.__make_offsets(
+            multi_hot_sizes, num_embeddings_per_feature, batch_size
+        )
+
+        # For plotting frequency access
+        self.collect_freqs_stats = collect_freqs_stats
+        self.model_to_track = None
+        self.freqs_pre_hash = []
+        self.freqs_post_hash = []
+        for embs_count in num_embeddings_per_feature:
+            self.freqs_pre_hash.append(np.zeros((embs_count)))
+            self.freqs_post_hash.append(np.zeros((embs_count)))
+
+    def save_freqs_stats(self) -> None:
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            rank = torch.distributed.get_rank()
+        else:
+            rank = 0
+        pre_dict = {str(k): e for k, e in enumerate(self.freqs_pre_hash)}
+        np.save(f"stats_pre_hash_{rank}_{self.dist_type}.npy", pre_dict)
+        post_dict = {str(k): e for k, e in enumerate(self.freqs_post_hash)}
+        np.save(f"stats_post_hash_{rank}_{self.dist_type}.npy", post_dict)
+
+    def pause_stats_collection_during_val_and_test(
+        self, model: torch.nn.Module
+    ) -> None:
+        self.model_to_track = model
+
+    def __make_multi_hot_indices_tables(
+        self,
+        dist_type: str,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+    ) -> List[np.array]:
+        np.random.seed(
+            0
+        )  # The seed is necessary for all ranks to produce the same lookup values.
+        multi_hot_tables_l = []
+        for embs_count, multi_hot_size in zip(
+            num_embeddings_per_feature, multi_hot_sizes
+        ):
+            embedding_ids = np.arange(embs_count)[:, np.newaxis]
+            if dist_type == "uniform":
+                synthetic_sparse_ids = np.random.randint(
+                    0, embs_count, size=(embs_count, multi_hot_size - 1)
+                )
+            elif dist_type == "pareto":
+                synthetic_sparse_ids = (
+                    np.random.pareto(
+                        a=0.25, size=(embs_count, multi_hot_size - 1)
+                    ).astype(np.int32)
+                    % embs_count
+                )
+            multi_hot_table = np.concatenate(
+                (embedding_ids, synthetic_sparse_ids), axis=-1
+            )
+            multi_hot_tables_l.append(multi_hot_table)
+        multi_hot_tables_l = [
+            torch.from_numpy(multi_hot_table).int()
+            for multi_hot_table in multi_hot_tables_l
+        ]
+        return multi_hot_tables_l
+
+    def __make_offsets(
+        self,
+        multi_hot_sizes: int,
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+    ) -> List[torch.Tensor]:
+        lS_o = torch.ones(
+            (len(num_embeddings_per_feature) * batch_size), dtype=torch.int32
+        )
+        for k, multi_hot_size in enumerate(multi_hot_sizes):
+            lS_o[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        lS_o = torch.cumsum(torch.concat((torch.tensor([0]), lS_o)), axis=0)
+        return lS_o
+
+    def __make_new_batch(
+        self,
+        lS_i: torch.Tensor,
+        batch_size: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        lS_i = lS_i.reshape(-1, batch_size)
+        multi_hot_ids_l = []
+        for k, (sparse_data_batch_for_table, multi_hot_table) in enumerate(
+            zip(lS_i, self.multi_hot_tables_l)
+        ):
+            multi_hot_ids = torch.nn.functional.embedding(
+                sparse_data_batch_for_table, multi_hot_table
+            )
+            multi_hot_ids = multi_hot_ids.reshape(-1)
+            multi_hot_ids_l.append(multi_hot_ids)
+            if self.collect_freqs_stats and (
+                self.model_to_track is None or self.model_to_track.training
+            ):
+                idx_pre, cnt_pre = np.unique(
+                    sparse_data_batch_for_table, return_counts=True
+                )
+                idx_post, cnt_post = np.unique(multi_hot_ids, return_counts=True)
+                self.freqs_pre_hash[k][idx_pre] += cnt_pre
+                self.freqs_post_hash[k][idx_post] += cnt_post
+        lS_i = torch.cat(multi_hot_ids_l)
+        if batch_size == self.batch_size:
+            return lS_i, self.offsets
+        else:
+            return lS_i, self.__make_offsets(
+                self.multi_hot_sizes, self.num_embeddings_per_feature, batch_size
+            )
+
+    def convert_to_multi_hot(self, batch: Batch) -> Batch:
+        batch_size = len(batch.dense_features)
+        lS_i = batch.sparse_features._values
+        lS_i, lS_o = self.__make_new_batch(lS_i, batch_size)
+        new_sparse_features = KeyedJaggedTensor.from_offsets_sync(
+            keys=batch.sparse_features._keys,
+            values=lS_i,
+            offsets=lS_o,
+        )
+        return Batch(
+            dense_features=batch.dense_features,
+            sparse_features=new_sparse_features,
+            labels=batch.labels,
+        )
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/requirements.txt b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/requirements.txt
new file mode 100644
index 000000000..270005d8d
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/requirements.txt
@@ -0,0 +1,4 @@
+fbgemm-gpu==0.3.2
+torchmetrics==0.11.0
+torchrec==0.3.2
+torchsnapshot
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py
new file mode 100644
index 000000000..2357451f4
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py
@@ -0,0 +1,2 @@
+from .generate_plan import generate_plan
+from .planner import Cost, CostModel, Planner, ShardingState
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py
new file mode 100644
index 000000000..46884b717
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py
@@ -0,0 +1,152 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from argparse import Namespace
+from itertools import chain
+from typing import List
+
+from .planner import CostModel, Planner
+
+
+def generate_plan(
+    slot_size_array: List[int],
+    multi_hot_sizes: List[int],
+    num_nodes: int,
+    num_gpus: int,
+    args: Namespace,
+    log_result: bool,
+):
+    def sanity_check(shard_matrix, shard_strategy):
+        # mainly to make sure all the tables are sharded
+        msg = "Not all tables covered in the sharding plan"
+        assert set(chain(*shard_matrix)) == set(range(len(slot_size_array))), msg
+        shard_strategy_list = [x for strategy_pair in shard_strategy for x in strategy_pair[1]]
+        assert set(shard_strategy_list) == set(range(len(slot_size_array))), msg
+
+        for table_list in shard_matrix:
+            if len(table_list) == 0:
+                raise Exception("Currently no empty shard list is allowed")
+
+    def int_to_string(shard_matrix_int, shard_strategy_int):
+        shard_strategy, shard_matrix = [], []
+        for pair in shard_strategy_int:
+            if len(pair[1]) != 0:
+                shard_strategy.append((pair[0], [str(x) for x in pair[1]]))
+        for sub_matrix_ in shard_matrix_int:
+            shard_matrix.append([str(x) for x in sub_matrix_])
+        return shard_matrix, shard_strategy
+
+    if args.sharding_plan in ["round_robin", "uniform"]:
+        # sharding strategies that don't exploit system configs
+        if args.sharding_plan == "round_robin":
+            mp_table = [i for i in range(len(slot_size_array))]
+            shard_matrix_ = [[] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in mp_table])]
+
+            for i, table_id in enumerate(mp_table):
+                target_gpu = i % num_gpus
+                shard_matrix_[target_gpu].append(table_id)
+
+        elif args.sharding_plan == "uniform":
+            shard_matrix_ = [[x for x in range(len(slot_size_array))] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in range(len(slot_size_array))])]
+
+    elif args.sharding_plan in ["auto", "hier_auto"]:
+        # sharding strategies that exploit system configs
+        dram_cap = args.memory_cap_for_embedding
+        if args.optimizer == "adagrad":
+            byte_per_elem = 8
+        elif args.optimizer == "sgd":
+            byte_per_elem = 4
+
+        if args.sharding_plan == "auto":
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_gpus, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_ = planner.plan()
+
+        elif args.sharding_plan == "hier_auto":
+            if num_nodes <= 1:
+                raise Exception("hier_auto plan is only applicable to configs with more than one node")
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap * args.num_gpus_per_node,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_nodes, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_node_ = planner.plan()
+            shard_matrix_ = []
+            for node_shard_matrix in shard_matrix_node_:
+                for i in range(args.num_gpus_per_node):
+                    shard_matrix_.append(node_shard_matrix)
+    elif args.sharding_plan in ["max_min"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[] for _ in range(num_gpus)]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+        dict_feature = {}
+        for k, v in enumerate(slot_size_array):
+            dict_feature[k] = v
+        dict_feature_list = sorted(dict_feature.items(),key=lambda x:x[1], reverse=True)
+        end = len(dict_feature_list) - 1
+        for i in range(len(dict_feature_list)):
+            index = i%num_gpus if int(i/num_gpus) == 0 else (num_gpus -1 - i%num_gpus)
+            shard_matrix_[index].append(dict_feature_list[i][0])
+            if (i >= end):
+                break
+            elif(end - i >= num_gpus - i%num_gpus):
+                shard_matrix_[index].append(dict_feature_list[end][0])
+                end -= 1
+    elif args.sharding_plan in ["custom"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[0, 5], [9, 16], [19, 12], [20, 18], [21, 25], [10, 8, 2, 13, 3], [22, 24, 4, 7, 23], [11, 15, 1, 17, 14, 6]]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+    else:
+        raise Exception("unknown sharding plan")
+
+    sanity_check(shard_matrix_, shard_strategy_)
+    shard_matrix, shard_strategy = int_to_string(shard_matrix_, shard_strategy_)
+
+    if log_result:
+        logging.info("Provided system info: ")
+        #logging.info("num_gpu_per_nodes: %d", args.num_gpus_per_node)
+        #logging.info("Memory to communication BW ratio: %f", args.mem_comm_bw_ratio)
+        #logging.info("Memory to communication work ratio: %f", args.mem_comm_work_ratio)
+        #logging.info("DRAM capacity: %f GB", args.memory_cap_for_embedding)
+        logging.info("shard_matrix:")
+        logging.info(shard_matrix)
+        logging.info("\n")
+
+    return shard_matrix, shard_strategy, shard_matrix_, shard_strategy_
diff --git a/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py
new file mode 100644
index 000000000..301171c17
--- /dev/null
+++ b/models/recommendation/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py
@@ -0,0 +1,313 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import time
+from typing import List, Tuple
+
+import numpy as np
+
+
+class ShardingState:
+    """
+    Containing the state of a sharding process.
+    The plan iteratively update the sharding state based on a given heuristic and obtain
+    solutions.
+    """
+
+    def __init__(
+        self,
+        array_hotness: np.array,
+        num_bucket: int,
+        dp_table_id: np.array(int) = np.array([]),
+    ) -> None:
+        mp_table_id = np.setdiff1d(np.arange(array_hotness.size), dp_table_id)
+        array_hotness_mp = array_hotness[mp_table_id]
+        sorted_idx = np.argsort(array_hotness_mp)[::-1]
+        self.array_unshard_hotness = array_hotness
+        self.array_hotness = array_hotness_mp[sorted_idx]
+        self.num_bucket = num_bucket
+        self.array_table_id = mp_table_id[sorted_idx]
+        self.array_num_split = np.zeros(self.array_unshard_hotness.size, dtype=int)
+        self.array_num_split[mp_table_id] = 1
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def split_hot_shard(self):
+        """
+        split the shard with the largest hotness
+        """
+        # shards are sorted based on the hotness. Find the first hot shard that
+        # can be split further
+        for shard_id in range(self.array_table_id.size):
+            table_id = self.array_table_id[shard_id]
+            hotness = self.array_unshard_hotness[table_id]
+            if self.array_num_split[table_id] * 2 <= self.num_bucket:
+                # if this table can be furter splitted and we can put it into
+                # more buckets
+                idx = np.where(self.array_table_id == table_id)[0]
+                self.array_hotness = np.delete(self.array_hotness, idx)
+                self.array_table_id = np.delete(self.array_table_id, idx)
+                self.array_num_split[table_id] *= 2
+                self.array_hotness = np.concatenate(
+                    (
+                        self.array_hotness,
+                        np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                    )
+                )
+                self.array_table_id = np.concatenate(
+                    (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+                )
+                break
+
+        # sort after splitting to maintain the shard hotness in order
+        sorted_idx = np.argsort(self.array_hotness)[::-1]
+        self.array_hotness = self.array_hotness[sorted_idx]
+        self.array_table_id = self.array_table_id[sorted_idx]
+
+    def split_oom_shard(self, table_id: int) -> bool:
+        hotness = self.array_unshard_hotness[table_id]
+        if self.array_num_split[table_id] * 2 <= self.num_bucket:
+            idx = np.where(self.array_table_id == table_id)[0]
+            self.array_hotness = np.delete(self.array_hotness, idx)
+            self.array_table_id = np.delete(self.array_table_id, idx)
+            self.array_num_split[table_id] *= 2
+            self.array_hotness = np.concatenate(
+                (
+                    self.array_hotness,
+                    np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                )
+            )
+            self.array_table_id = np.concatenate(
+                (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+            )
+            sorted_idx = np.argsort(self.array_hotness)[::-1]
+            self.array_hotness = self.array_hotness[sorted_idx]
+            self.array_table_id = self.array_table_id[sorted_idx]
+            return True
+        else:
+            return False
+
+    def update_split_num(self):
+        self.array_num_split = np.zeros_like(self.array_unshard_hotness)
+        for shard_list in self.shard_ll:
+            for table_id in shard_list:
+                self.array_num_split[table_id] += 1
+
+    def reset_shard_ll(self):
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def push_bucket(
+        self,
+        bucket_id: int,
+        table_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].append(table_id)
+
+    def pop_bucket(
+        self,
+        bucket_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].pop()
+
+
+class Cost:
+    def __init__(
+        self,
+        cost: np.array(float),
+        hotness_cost: np.array(float),
+        table_cost: np.array(float),
+        mem_cost: np.array(float),
+    ) -> None:
+        self.cost = cost
+        self.hotness_cost = hotness_cost
+        self.table_cost = table_cost
+        self.mem_cost = mem_cost
+
+
+class CostModel:
+    def __init__(
+        self,
+        hotness_cost: float,
+        table_cost: float,
+        mem_cost: float,
+        mem_capacity: float,
+        table_size: List[int],
+    ) -> None:
+        self.unit_hotness_cost = hotness_cost
+        self.unit_table_cost = table_cost
+        self.unit_mem_cost = mem_cost
+        self.mem_capacity = mem_capacity
+        self.array_table_size = np.array(table_size)
+
+    def get_cost(
+        self,
+        ss: ShardingState,
+    ) -> Tuple[Cost, bool]:
+        list_cost = []
+        list_hotness_cost = []
+        list_table_cost = []
+        list_mem_cost = []
+
+        for shard_list in ss.shard_ll:
+            hotness_cost = (
+                self.unit_hotness_cost
+                * (ss.array_unshard_hotness[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            table_cost = self.unit_table_cost * len(shard_list)
+            mem_cost = (
+                self.unit_mem_cost
+                * (self.array_table_size[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            list_cost.append(hotness_cost + table_cost)
+            list_hotness_cost.append(hotness_cost)
+            list_table_cost.append(table_cost)
+            list_mem_cost.append(mem_cost)
+
+        return (
+            Cost(np.array(list_cost), np.array(list_hotness_cost), np.array(list_table_cost), np.array(list_mem_cost)),
+            max(list_mem_cost) > self.mem_capacity,
+        )
+
+    def deduct_mem_cap_for_dp(
+        self,
+        dp_table_id: list,
+    ) -> None:
+        self.mem_capacity -= self.array_table_size[dp_table_id].sum() * self.unit_mem_cost
+        if self.mem_capacity < 0:
+            raise Exception("OOM due to DP. Please considering increase the DP threshold")
+
+
+class Planner:
+    """
+    The planner work out a series of plans iteratively.
+    In each iteration, the planner tries to split the hottest shard and place the shards into
+    a bucket based on a give heuristic. When the shard is too large to fit into the best bucket
+    suggested by the heuristic, it finds the next best bucket until it iterates through all the
+    buckets. In that case, it tries to split the shard further. If the shard cannot be split
+    further, the planner aborts and returns the default sharding plan.
+    """
+
+    def __init__(
+        self,
+        list_hotness: list,
+        num_bucket: int,
+        cost_model: CostModel,
+        dp_threshold: int = 0,
+        max_search_iter: int = 20,
+        log_result: bool = False,
+    ) -> None:
+        self.array_hotness = np.array(list_hotness)
+        self.num_bucket = num_bucket
+        self.cost_model = cost_model
+        self.list_candidate = []
+        self.max_search_iter = max_search_iter
+        self.log_result = log_result
+
+        # Create the default sharding plan. Throw if even this default sharding plan cannot fit, as
+        # it should be the most memory-efficient
+        sharding_state_default = ShardingState(self.array_hotness, self.num_bucket)
+        for b in range(self.num_bucket):
+            for t in range(self.array_hotness.size):
+                sharding_state_default.push_bucket(b, t)
+        sharding_state_default.update_split_num()
+        cost, oom = self.cost_model.get_cost(sharding_state_default)
+        if oom:
+            raise Exception("OOM even with the most memory-efficient sharding plan")
+        self.list_candidate.append(
+            (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, sharding_state_default.shard_ll)
+        )
+
+        # Create DP sharding plan based on the DP threshold
+        self.dp_table_id = np.where(cost_model.array_table_size < dp_threshold / cost_model.unit_mem_cost)[0]
+        self.mp_table_id = np.setdiff1d(np.arange(self.array_hotness.size), self.dp_table_id)
+        self.sharding_state = ShardingState(self.array_hotness, self.num_bucket, self.dp_table_id)
+        self.cost_model.deduct_mem_cap_for_dp(self.dp_table_id)
+
+        logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    def greedy_plan(self, ss):
+        """
+        This is a heuristic based on greedy policy. The shard is placed to the bucket with the
+        lowest hotness cost
+        """
+        array_cost = np.zeros(ss.num_bucket)
+        ss.reset_shard_ll()
+        for i in range(ss.array_hotness.size):
+            sorted_idx = np.argsort(array_cost)
+            sharded = False
+            for bucket_id in sorted_idx:
+                if ss.array_table_id[i] not in ss.shard_ll[bucket_id]:
+                    # for now, only uniform sharding is supported. Hence cannot put two shards
+                    # from the same table into the same bucket
+                    ss.push_bucket(bucket_id, ss.array_table_id[i])
+                    cost, oom = self.cost_model.get_cost(ss)
+                    if not oom:
+                        sharded = True
+                        array_cost = cost.cost
+                        break
+                    else:
+                        # Currnet bucket cannot fit. Iterate to the next best bucket
+                        ss.pop_bucket(bucket_id)
+            if not sharded:
+                # This means the shard is too large to fit within any bucket
+                return ss.array_table_id[i], ss, cost
+        return None, ss, cost
+
+    def plan(self):
+        t0 = time.time()
+        for i in range(self.max_search_iter):
+            oom_table_id, self.sharding_state, cost = self.greedy_plan(self.sharding_state)
+            if oom_table_id is None:
+                self.list_candidate.append(
+                    (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, self.sharding_state.shard_ll)
+                )
+                self.sharding_state.split_hot_shard()
+            else:
+                oom_table_can_split = self.sharding_state.split_oom_shard(oom_table_id)
+                if not oom_table_can_split:
+                    break
+
+        self.list_candidate.sort(key=lambda x: x[0])
+
+        shard_strategy = [("mp", self.mp_table_id.tolist())]
+        shard_strategy.append(("dp", self.dp_table_id.tolist()))
+        shard_matrix = self.list_candidate[0][-1]
+        for table_id in self.dp_table_id:
+            for shard_list in shard_matrix:
+                shard_list.append(table_id)
+        if self.log_result:
+            logging.info("Planner took %f sec" % (time.time() - t0))
+            logging.info(shard_strategy)
+            logging.info(shard_matrix)
+            logging.info("hotness cost is:")
+            logging.info(self.list_candidate[0][1])
+            logging.info("table cost is:")
+            logging.info(self.list_candidate[0][2])
+            logging.info("mem cost is:")
+            logging.info(self.list_candidate[0][3])
+        return shard_strategy, shard_matrix
diff --git a/notebooks/AI_Reference_Models.ipynb b/notebooks/AI_Reference_Models.ipynb
new file mode 100644
index 000000000..0257c5973
--- /dev/null
+++ b/notebooks/AI_Reference_Models.ipynb
@@ -0,0 +1,817 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#  Intel® AI Reference Models Jupyter Notebook\n",
+    "\n",
+    "This Jupyter notebook helps you choose and run a comparison between two models from the [Intel® AI Reference Models repo](https://github.com/IntelAI/models) using Intel® Optimizations for TensorFlow*. When you run the notebook, it installs required package dependencies, displays information about your platform, lets you choose the two models to compare, runs those models, and finally displays a performance comparison chart. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id='section_1'></a>\n",
+    "# Step 1: Display Platform Information "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Install Tensorflow and required dependencies for the jupyter notebook"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install matplotlib ipykernel psutil pandas cxxfilt gitpython \n",
+    "!pip install intel-tensorflow\n",
+    "!pip install gcg\n",
+    "!python3 -m pip install gitpython\n",
+    "!pip install prettytable\n",
+    "!pip install --upgrade matplotlib"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "Set the path for the AI reference models, assumed to be the current working directory, and the path for the utility functions directory inside the cloned Intel® AI Reference Models directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "# If default path does not work, change AIReferenceRoot path according to your environment\n",
+    "current_path = os.getcwd()\n",
+    "os.environ['AIReferenceRoot'] = os.path.dirname(current_path) + \"/\"\n",
+    "os.environ['ProfileUtilsRoot'] = os.environ['AIReferenceRoot'] + \"notebooks/profiling/\"\n",
+    "print(\"Path for the AI Reference Models root is: \", os.environ['AIReferenceRoot'])\n",
+    "print(\"Path for utility functions directory is: \", os.environ['ProfileUtilsRoot'])\n",
+    "\n",
+    "# Check for mandatory python scripts after AIReferenceRoot and ProfileUtilsRoot are assigned\n",
+    "import os\n",
+    "current_path = os.getcwd()\n",
+    "benchmark_path = os.environ['AIReferenceRoot'] + \"benchmarks/launch_benchmark.py\"\n",
+    "if os.path.exists(benchmark_path) == True:\n",
+    "    print(benchmark_path)\n",
+    "else:\n",
+    "    print(\"ERROR! Can't find benchmark/launch_benchmark.py script!\")\n",
+    "\n",
+    "profile_utils_path = os.environ['ProfileUtilsRoot'] + \"profile_utils.py\"\n",
+    "if os.path.exists(profile_utils_path) == True:\n",
+    "    print(profile_utils_path)\n",
+    "else:\n",
+    "    print(\"ERROR! Can't find profile_utils.py script!\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Print information about the platform specs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from profiling.profile_utils import PlatformUtils\n",
+    "plat_utils = PlatformUtils()\n",
+    "plat_utils.dump_platform_info()\n",
+    "\n",
+    "\n",
+    "# Import AI Reference models CPU info\n",
+    "import sys\n",
+    "sys.path.append(os.environ['AIReferenceRoot']+os.sep+'benchmarks/common/')\n",
+    "from platform_util import PlatformUtil \n",
+    "cpu_info = PlatformUtil(\"\")\n",
+    "\n",
+    "# Print Tensorflow version\n",
+    "import tensorflow as tf\n",
+    "print (\"We are using Tensorflow version\", tf.__version__)\n",
+    "\n",
+    "# Display the CPU info\n",
+    "import os\n",
+    "numa_nodes = cpu_info.numa_nodes\n",
+    "print(\"CPU count per socket:\" , cpu_info.cores_per_socket ,\" \\nSocket count:\", cpu_info.sockets, \" \\nNuma nodes:\",numa_nodes)\n",
+    "if numa_nodes > 0:\n",
+    "    socket_number = 1\n",
+    "    cpu_count = cpu_info.cores_per_socket\n",
+    "    inter_thread = 1\n",
+    "else:\n",
+    "    # on non-numa machine, we should use all the cores and don't use numactl\n",
+    "    socket_number = -1\n",
+    "    cpu_count = cpu_info.cores_per_socket * cpu_info.sockets\n",
+    "    inter_thread = cpu_info.sockets"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 2: Run the first model\n",
+    "\n",
+    "This notebook helps you compare the performance of two models listed in the supported models. Select the first model to run and compare.  (If the environment variable MODEL_1_INDEX is set, we'll use that instead of prompting for input.)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.1: Display all the supported models and select first  model to run."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "After the list of models is displayed, select one by entering the model index number."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from profiling.profile_utils import AIReferenceConfigFile\n",
+    "from prettytable import PrettyTable\n",
+    "\n",
+    "config = AIReferenceConfigFile()\n",
+    "sections = config.read_supported_section()\n",
+    "\n",
+    "# Create a table with headers\n",
+    "models_table = PrettyTable([\"Index\", \"Model Name\", \"Framework\", \"Mode\", \"Script Type\", \"Precision\"])\n",
+    "\n",
+    "# Iterate through the sections and add rows to the models table:\n",
+    "for index, section in enumerate(sections):\n",
+    "    split_section = section.split()\n",
+    "\n",
+    "    if len(split_section) >= 5:\n",
+    "        modelname = split_section[0]\n",
+    "        framework = split_section[1]\n",
+    "        mode = split_section[2]\n",
+    "        script_type = split_section[3]\n",
+    "        precision = split_section[4]\n",
+    "    \n",
+    "        models_table.add_row([index, modelname, framework, mode, script_type, precision])\n",
+    "\n",
+    "print(\"Supported Models: \")\n",
+    "print(models_table)\n",
+    "\n",
+    "# use the \"MODEL_1_INDEX\" environment variable value if it exists.\n",
+    "import os\n",
+    "env_model_1_index=os.environ.get('MODEL_1_INDEX', '')\n",
+    "if env_model_1_index != '':\n",
+    "    model_1_index= int(env_model_1_index)\n",
+    "else:\n",
+    "    ## USER INPUT\n",
+    "    model_1_index= int(input('Input a index number of a model: '))\n",
+    "\n",
+    "# List out the selected model name\n",
+    "if model_1_index >= len(sections):\n",
+    "    print(\"ERROR! Input a model_index value between 0 and \", len(sections))\n",
+    "else:\n",
+    "    model_1_name=sections[model_1_index]\n",
+    "\n",
+    "# Prints out model name\n",
+    "print(\"First model is: \", model_1_name)\n",
+    "\n",
+    "# Set the environment variable for precision to run the quickstart scripts\n",
+    "configvals = []\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "configvals=config.read_config(model_1_name)\n",
+    "os.environ['PRECISION']=config.precision"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.2: Get the required dataset for the selected model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The following cell checks the dataset path. If the data-location is already specified, then the notebook will use the dataset path mentioned in ai_reference_models.ini file. If the data-location is not specified in the ai_reference_models.ini, the data downloading option instructions are shown. You must manually download the dataset using these instructions before you can proceed to the next cell."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "# Get the parameters from config\n",
+    "config = AIReferenceConfigFile()\n",
+    "config.read_config(model_1_name)\n",
+    "data_download_path=''\n",
+    "model_source_dir=''\n",
+    "if config.data_download != '': #and config.data_location == '':\n",
+    "    print(\"\\nFollow these instructions to get the data : \")\n",
+    "    if config.data_download != '':\n",
+    "        val = config.read_value_from_section(model_1_name, 'data-download')\n",
+    "    print(val)\n",
+    "    # use the \"DATA_DOWNLOAD_PATH\" environment variable value if it exists.\n",
+    "    env_data_download_path=os.environ.get('DATA_DOWNLOAD_PATH', '')\n",
+    "    if env_data_download_path != '':\n",
+    "        data_download_path= env_data_download_path"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Set the path for dataset directory for the first model:\n",
+    "\n",
+    "DATASET_DIR': the path where the dataset exists and is downloaded.\n",
+    "\n",
+    "**ACTION : You need to input the path where the dataset for the first model exists or where you have downloaded it in your system**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_path = input('Input the path where the dataset exists for the first model:')\n",
+    "\n",
+    "os.environ['DATASET_DIR'] = dataset_path\n",
+    "print(\"Data location path:\", os.environ['DATASET_DIR'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.3: Prepare pre-trained model for the selected model\n",
+    "\n",
+    "This step checks if the pre-trained model for the selected model exists in the pre-trained directory path. If the pre-trained directory does not exist, then it downloads the pre-trained model for the selected precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "config = AIReferenceConfigFile()\n",
+    "configvals = []\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "configvals=config.read_config(model_1_name)\n",
+    "\n",
+    "# Get the pre-trained model file\n",
+    "if config.wget != '':\n",
+    "    pretrain_model_path = config.download_pretrained_model(current_path=current_path)\n",
+    "    pretrain_model_path = config.uncompress_file(pretrain_model_path, current_path=current_path)\n",
+    "\n",
+    "# Add custom arguments\n",
+    "if config.custom_args != '':\n",
+    "    configvals.append(\"--\")\n",
+    "    custom_config = config.parsing_custom_args(model_1_name, config.custom_args)\n",
+    "    configvals = configvals + custom_config\n",
+    "    \n",
+    "# Combine common parameters and config parameters\n",
+    "params = configvals    \n",
+    "    \n",
+    "sys.argv=[benchmark_path]+params\n",
+    "print(sys.argv)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Set the environment variable for pre-trained model for the first model:\n",
+    "\n",
+    "'PRETRAINED_MODEL': the path where is the pretrained_model exists and is downloaded.\n",
+    "\n",
+    "NOTE: You can change the value of 'PRETRAINED_MODEL' by changing its assignment in the cell below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"PRETRAINED_MODEL\"]=pretrain_model_path\n",
+    "print(\"Pretrain_model_path:\", os.environ[\"PRETRAINED_MODEL\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.4: Set the log directory or output directory\n",
+    "\n",
+    "'OUTPUT_DIR': the output directory path where model output logs are collected.\n",
+    "\n",
+    "The default directory name for the output logs is \\\"logs\\\" in the current working directory. You can change this directory name by replacing the value assigned to log_directory in the cell below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set the log directory/output directory to store the logs\n",
+    "log_directory=os.getcwd() + os.sep + \"logs\"\n",
+    "print(log_directory)\n",
+    "\n",
+    "#Set output-dir directory\n",
+    "if log_directory !='':\n",
+    "    configvals.append(\"--output-dir\")\n",
+    "    configvals.append(log_directory)\n",
+    "\n",
+    "os.environ['OUTPUT_DIR']=log_directory\n",
+    "print(\"Output directory path is:\", os.environ['OUTPUT_DIR'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.5:  Run the first Model "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**OPTIONAL: You can change the batch size from the model's default. For online_inference,  set batch_size value to be 1.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "batch_size= int(input('Set the value for batch size that you want to run: '))\n",
+    "os.environ[\"BATCH SIZE\"]=batch_size"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the first model using the quickstart script configured in the ai_reference_models.ini file, and save output logs to the selected log directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = AIReferenceConfigFile()\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "config.read_config(model_1_name)\n",
+    "\n",
+    "# Split the model_name into individual parts\n",
+    "parts = model_1_name.split()\n",
+    "\n",
+    "# Join the parts using hyphens as the separator\n",
+    "log_name = '-'.join(parts)\n",
+    "log_name = log_name + \".log\"\n",
+    "\n",
+    "ai_reference_root = os.environ.get('AIReferenceRoot')\n",
+    "\n",
+    "%cd $ai_reference_root\n",
+    "run_workload = (\"quickstart/\" + config.ai_type + \"/\" + config.framework + \"/\"+ config.model_name + \"/\"\n",
+    "                + config.mode + \"/\" + config.device +\"/\" + config.script ) \n",
+    "\n",
+    "!./$run_workload | tee $log_directory/{log_name}\n",
+    "\n",
+    "%cd -"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2.6 Get the throughput or accuracy of the first model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Throughput of first workload:\n",
+    "# Split the model_name into individual parts\n",
+    "parts_1 = model_1_name.split()\n",
+    "\n",
+    "# Join the parts using hyphens as the separator\n",
+    "log_1_name = '-'.join(parts_1)\n",
+    "log_1_name = log_1_name + \".log\"\n",
+    "\n",
+    "!grep -A 1 \"Throughput summary:\" $log_directory/{log_1_name} | tail -n 1\n",
+    "\n",
+    "import subprocess\n",
+    "# Run the grep command and capture its output\n",
+    "try:\n",
+    "    grep_output = subprocess.check_output(f'grep -A 1 \"Throughput summary:\" {log_directory}/{log_1_name} | tail -n 1; grep -A 1 \"Summary total images/sec:\" {log_directory}/{log_1_name} | tail -n 1', shell=True, universal_newlines=True)\n",
+    "except subprocess.CalledProcessError:\n",
+    "    grep_output = \"Pattern not found in the file.\"\n",
+    "\n",
+    "# Print or use the captured output as needed\n",
+    "print(\"Throughput of \", model_1_name, \": \", grep_output.strip())  # Remove leading/trailing whitespace\n",
+    "\n",
+    "# Store the captured output in a variable for further use\n",
+    "Throughput_of_workload_1 = grep_output.strip()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 3: Select and run the second model\n",
+    "\n",
+    "Let's now run a second model and compare its performance with the first."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3.1 Select another model for comparision\n",
+    "\n",
+    "After the list of models is displayed, select one by entering the model index number. (If the environment variable MODEL_2_INDEX is set, we'll use that instead of prompting for input.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Supported Models: \")\n",
+    "print(models_table)\n",
+    "\n",
+    "# use the \"MODEL_2_INDEX\" environment variable value if it exists.\n",
+    "env_model_2_index=os.environ.get('MODEL_2_INDEX', '')\n",
+    "if env_model_2_index != '':\n",
+    "    model_2_index= int(env_model_2_index)\n",
+    "else:\n",
+    "    ## USER INPUT\n",
+    "    model_2_index= int(input('Input a index number of second model: '))\n",
+    "\n",
+    "# List out the selected model name\n",
+    "if model_2_index >= len(sections):\n",
+    "    print(\"ERROR! Input a model_index value between 0 and \", len(sections))\n",
+    "else:\n",
+    "    model_2_name=sections[model_2_index]\n",
+    "\n",
+    "# Prints out model name\n",
+    "print(\"Second model is: \", model_2_name)\n",
+    "\n",
+    "# Set the environment variable for precision to run the quickstart scripts\n",
+    "configvals = []\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "configvals=config.read_config(model_2_name)\n",
+    "os.environ['PRECISION']=config.precision"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3.2: Get the required dataset for the selected model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The following cell checks for the second dataset path. If the data-location is already specified, then the notebook will use the dataset path mentioned in ai_reference_models.ini file. If the data-location is not specified in the ai_reference_models.ini, the data downloading option instructions are shown. You must manually download the dataset using these instructions before you can proceed to the next cell."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get the parameters from config\n",
+    "config = AIReferenceConfigFile()\n",
+    "config.read_config(model_2_name)\n",
+    "data_download_path=''\n",
+    "model_source_dir=''\n",
+    "if config.data_download != '' and config.data_location == '':\n",
+    "    print(\"\\nFollow these instructions to get the data : \")\n",
+    "    if config.data_download != '':\n",
+    "        val = config.read_value_from_section(model_2_name, 'data-download')\n",
+    "    print(val)\n",
+    "    # use the \"DATA_DOWNLOAD_PATH\" environment variable value if it exists.\n",
+    "    env_data_download_path=os.environ.get('DATA_DOWNLOAD_PATH', '')\n",
+    "    if env_data_download_path != '':\n",
+    "        data_download_path= env_data_download_path"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Set the path for dataset directory for the second model:\n",
+    "\n",
+    "DATASET_DIR': the path where the dataset exists and is downloaded.\n",
+    "\n",
+    "**ACTION : You need to input the path where the dataset for the second model exists or where you have downloaded it in your system**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_path = input('Input the path where the dataset exists for the second model:')\n",
+    "\n",
+    "os.environ['DATASET_DIR'] = dataset_path\n",
+    "print(\"Data location path:\", os.environ['DATASET_DIR'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3.3: Prepare pre-trained model for the selected model\n",
+    "\n",
+    "This step checks if the pre-trained model for the selected model exists in the pre-trained directory path. If the pre-trained directory does not exist, then it downloads the pre-trained model for the selected precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = AIReferenceConfigFile()\n",
+    "configvals = []\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "configvals=config.read_config(model_2_name)\n",
+    "\n",
+    "# Set the log directory/output directory to store the logs\n",
+    "log_directory=os.getcwd() + os.sep + \"logs\"\n",
+    "print(log_directory)\n",
+    "\n",
+    "# Get the pre-trained model file\n",
+    "if config.wget != '':\n",
+    "    pretrain_model_path = config.download_pretrained_model(current_path=current_path)\n",
+    "    pretrain_model_path = config.uncompress_file(pretrain_model_path, current_path=current_path)\n",
+    "    \n",
+    "#Set output-dir directory\n",
+    "if log_directory !='':\n",
+    "    configvals.append(\"--output-dir\")\n",
+    "    configvals.append(log_directory)\n",
+    "\n",
+    "# Add custom arguments\n",
+    "if config.custom_args != '':\n",
+    "    configvals.append(\"--\")\n",
+    "    custom_config = config.parsing_custom_args(model_2_name, config.custom_args)\n",
+    "    configvals = configvals + custom_config\n",
+    "    \n",
+    "# Combine common parameters and config parameters\n",
+    "params = configvals    \n",
+    "    \n",
+    "sys.argv=[benchmark_path]+params\n",
+    "print(sys.argv)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Set the environment variable for pre-trained model for the second model:\n",
+    "\n",
+    "'PRETRAINED_MODEL': the path where is the pretrained_model exists and is downloaded.\n",
+    "\n",
+    "NOTE: You can change the value of 'PRETRAINED_MODEL' by changing its assignment in the cell below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"PRETRAINED_MODEL\"]=pretrain_model_path\n",
+    "print(\"Pretrain_model_path:\", os.environ[\"PRETRAINED_MODEL\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3.4: Run the second model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**OPTIONAL: You can change the batch size from the model's default. For online_inference,  set batch_size value to be 1.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "batch_size= int(input('Set the value for batch size that you want to run: '))\n",
+    "os.environ[\"BATCH SIZE\"]=batch_size"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the second model using the quickstart script configured in the ai_reference_models.ini file, and save output logs to the selected log directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = AIReferenceConfigFile()\n",
+    "\n",
+    "# Get the parameters from config\n",
+    "config.read_config(model_2_name)\n",
+    "\n",
+    "# Split the model_name into individual parts\n",
+    "parts = model_2_name.split()\n",
+    "\n",
+    "# Join the parts using hyphens as the separator\n",
+    "log_name = '-'.join(parts)\n",
+    "log_name = log_name + \".log\"\n",
+    "\n",
+    "ai_reference_root = os.environ.get('AIReferenceRoot')\n",
+    "\n",
+    "%cd $ai_reference_root\n",
+    "run_workload = (\"quickstart/\" + config.ai_type + \"/\" + config.framework + \"/\"+ config.model_name + \"/\"\n",
+    "                + config.mode + \"/\" + config.device +\"/\" + config.script ) \n",
+    "\n",
+    "!./$run_workload | tee $log_directory/{log_name}\n",
+    "\n",
+    "%cd -"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3.5 Get the throughput or accuracy of the second model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "# Throughput of second workload:\n",
+    "# Split the model_name into individual parts\n",
+    "parts_2 = model_2_name.split()\n",
+    "\n",
+    "# Join the parts using hyphens as the separator\n",
+    "log_2_name = '-'.join(parts_2)\n",
+    "log_2_name = log_2_name + \".log\"\n",
+    "\n",
+    "!grep -A 1 \"Throughput summary:|Summary total images/sec:\" $log_directory/{log_2_name} | tail -n 1\n",
+    "\n",
+    "# Run the grep command and capture its output\n",
+    "try:\n",
+    "    grep_output = subprocess.check_output(f'grep -A 1 \"Throughput summary:\" {log_directory}/{log_2_name} | tail -n 1; grep -A 1 \"Summary total images/sec:\" {log_directory}/{log_2_name} | tail -n 1', shell=True, universal_newlines=True)\n",
+    "except subprocess.CalledProcessError:\n",
+    "    grep_output = \"Pattern not found in the file.\"\n",
+    "\n",
+    "# Print or use the captured output as needed\n",
+    "print(\"Throughput of \", model_2_name, \": \", grep_output.strip())  # Remove leading/trailing whitespace\n",
+    "\n",
+    "# Store the captured output in a variable for further use\n",
+    "Throughput_of_workload_2 = grep_output.strip()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 4: Compare performance results and plot a comparison chart for the two models"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Get the results (throughput/accuracy) of the two models for comparision"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Throughput of\", model_1_name, \": \", Throughput_of_workload_1)\n",
+    "print(\"Throughput of\", model_2_name, \": \", Throughput_of_workload_2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Plot a chart for comparision "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Define your data\n",
+    "categories = [model_1_name, model_2_name]\n",
+    "values = [Throughput_of_workload_1, Throughput_of_workload_2]\n",
+    "\n",
+    "# Generate a list of colors for each bar\n",
+    "colors = ['blue', 'orange']\n",
+    "\n",
+    "bars = plt.scatter(model_1_name,Throughput_of_workload_1 , label=model_1_name, color='blue')\n",
+    "bars = plt.scatter(model_2_name,Throughput_of_workload_2 , label=model_2_name, color='orange')\n",
+    "\n",
+    "# Add labels and a title\n",
+    "plt.xlabel('Workloads')\n",
+    "plt.ylabel('Throughput')\n",
+    "plt.title('Bar Chart of Two workloads')\n",
+    "\n",
+    "# Set the x-axis ticks and labels\n",
+    "plt.xticks(range(len(categories)), [])\n",
+    "\n",
+    "plt.legend()\n",
+    "\n",
+    "plt.gca().invert_yaxis()\n",
+    "\n",
+    "# Display the chart\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/notebooks/README.md b/notebooks/README.md
new file mode 100755
index 000000000..74159cb6f
--- /dev/null
+++ b/notebooks/README.md
@@ -0,0 +1,59 @@
+# Run Intel® AI Reference Models in a Jupyter Notebook
+ 
+This Jupyter notebook helps you choose and run a comparison between two models from the [Intel® AI Reference Models repo](https://github.com/IntelAI/models) using Intel® Optimizations for TensorFlow*. When you run the notebook, it installs required package dependencies, displays information about your platform, lets you choose the two models to compare, runs those models, and finally displays a performance comparison chart.
+  
+## Supported Models
+
+| Model | Framework | Mode | Script Type | Supported Precisions |
+| ----- | --------- | ---- | ----------- | -------------------- |
+| ResNet50v1.5 | TensorFlow | Inference | throughput_multi_instance | [FP32, BFloat32, FP16, INT8, BFloat16](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu) |
+| ResNet50v1.5 | TensorFlow | Inference | accuracy | [FP32, BFloat32, FP16, INT8, BFloat16](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu) |
+| ResNet50v1.5 | TensorFlow | Inference | realtime_weightsharing | [INT8, BFloat16](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu) |
+| ResNet50 | TensorFlow | Inference | batch_inference | [FP32, INT8](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu) |
+| ResNet50 | TensorFlow | Inference | accuracy | [FP32, INT8](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu) |
+| Mobilenet_v1 | TensorFlow | Inference | throughput_multi_instance | [FP32, BFloat32, INT8, BFloat16](/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu) |
+| MobileNet_v1 | TensorFlow | Inference | accuracy | [FP32, BFloat32, INT8, BFloat16](/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu) |
+| SSD-MobileNet | TensorFlow | Inference | throughput_multi_instance | [FP32, BFloat32, INT8, BFloat16](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu) |
+| SSD-MobileNet | TensorFlow | Inference | accuracy | [FP32, BFloat32, INT8, BFloat16](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu) |
+
+## Environment Setup
+
+Instead of installing or updating packages system-wide, it's a good idea to install project-specific Python packages in a Python virtual environment localized to your project. The Python virtualenv package lets you do just that.  Using virtualenv is optional, but recommended.
+
+The jupyter notebook runs on Ubuntu distribution for Linux.
+
+ 1. **Virtualenv Python Environment**
+       Install virtualenv on Ubuntu using these commands:
+       ```
+       sudo apt-get update
+       sudo apt-get install python-dev python-pip
+       sudo pip install -U virtualenv  # system-wide install
+       ```
+
+       Activate virtual environment using the following command:
+       ```
+       virtualenv -p python ai_ref_models
+       source ai_ref_models/bin/activate
+       ``` 
+
+ 2. **Jupyter Notebook Support**:
+
+       Install Jupyter notebook support with the command:
+       ```
+          pip install notebook
+       ```
+       Refer to the [Installing Jupyter guide](https://jupyter.org/install) for details.
+
+
+## How to Run the Notebook
+
+1. Clone the Intel® AI Reference Models repo: 
+   ```
+   git clone https://github.com/IntelAI/models.git
+   ```
+2. Launch the Jupyter notebook server: `jupyter notebook --ip=0.0.0.0`
+3. Follow the instructions to open the URL with the token in your browser, something like this: ` http://127.0.0.1:8888/tree?token=677d3ab5efd636530e9a695fb749511c6b669ad015cdb7ad`
+4. Browse to the `models/notebooks/` folder
+5. Click the AI_Reference_Models notebook file - [AI_Reference_Models.ipynb](https://github.com/IntelAI/models/notebooks/AI_Reference_Models.ipnyb).
+6. Read the instructions and run through each notebook cell, in order, ending with a display of the analysis results. Note that some cells prompt you for input, such as selecting the model number you'd like to run.
+7. When done, you should deactivate the virtualenv, if you used one, with the command: `deactivate`
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/fp32/__init__.py b/notebooks/profiling/__init__.py
similarity index 92%
rename from models/image_recognition/tensorflow/vision_transformer/inference/fp32/__init__.py
rename to notebooks/profiling/__init__.py
index ce0a27f82..4e2e67b6a 100644
--- a/models/image_recognition/tensorflow/vision_transformer/inference/fp32/__init__.py
+++ b/notebooks/profiling/__init__.py
@@ -1,7 +1,7 @@
 #
 # -*- coding: utf-8 -*-
 #
-# Copyright (c) 2023 Intel Corporation
+# Copyright (c) 2021-2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,4 +17,3 @@
 #
 
 #
-
diff --git a/notebooks/profiling/ai_reference_models.ini b/notebooks/profiling/ai_reference_models.ini
new file mode 100755
index 000000000..9126b16ea
--- /dev/null
+++ b/notebooks/profiling/ai_reference_models.ini
@@ -0,0 +1,383 @@
+[resnet50v1_5 tensorflow inference throughput_multi_instance fp32]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_throughput_multi_instance.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = Get the Imagenet dataset by following the instructions at "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference throughput_multi_instance bfloat32]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = Get the Imagenet dataset by following the instructions at "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference throughput_multi_instance fp16]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference throughput_multi_instance int8]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bias_resnet50.pb
+
+[resnet50v1_5 tensorflow inference throughput_multi_instance bfloat16]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bf16_resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference accuracy fp32]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference accuracy bfloat32]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference accuracy fp16]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://zenodo.org/record/2535873/files/resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference accuracy int8]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bias_resnet50.pb
+
+[resnet50v1_5 tensorflow inference accuracy bfloat16]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bf16_resnet50_v1.pb
+
+[resnet50v1_5 tensorflow inference realtime_weightsharing int8]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_realtime_weightsharing.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bias_resnet50.pb
+
+[resnet50v1_5 tensorflow inference realtime_weightsharing bfloat16]
+ai-type = image_recognition
+model-name = resnet50v1_5
+script = inference_realtime_weightsharing.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/2_8/bf16_resnet50_v1.pb
+
+[resnet50 tensorflow inference batch_inference int8]
+ai-type = image_recognition
+model-name = resnet50
+script = batch_inference.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_int8_pretrained_model.pb
+
+[resnet50 tensorflow inference batch_inference fp32]
+ai-type = image_recognition
+model-name = resnet50
+script = batch_inference.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_fp32_pretrained_model.pb
+
+[resnet50 tensorflow inference accuracy int8]
+ai-type = image_recognition
+model-name = resnet50
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_int8_pretrained_model.pb
+
+[resnet50 tensorflow inference accuracy fp32]
+ai-type = image_recognition
+model-name = resnet50
+script = accuracy.sh 
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_fp32_pretrained_model.pb
+
+[ssd-mobilenet tensorflow inference throughput_multi_instance fp32]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference throughput_multi_instance bfloat16]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference throughput_multi_instance bfloat32]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference throughput_multi_instance int8]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
+
+[ssd-mobilenet tensorflow inference accuracy fp32]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference accuracy bfloat16]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference accuracy bfloat32]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
+
+[ssd-mobilenet tensorflow inference accuracy int8]
+ai-type = object_detection
+model-name = ssd-mobilenet
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/SSDMobilenet
+data-download = To get the dataset, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/coco/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
+
+[mobilenet_v1 tensorflow inference throughput_multi_instance fp32]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference throughput_multi_instance bfloat16]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference throughput_multi_instance bfloat32]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference throughput_multi_instance int8]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = inference_throughput_multi_instance.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenetv1_int8_pretrained_model.pb
+
+[mobilenet_v1 tensorflow inference accuracy fp32]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = fp32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference accuracy bfloat16]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat16
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference accuracy bfloat32]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = bfloat32
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenet_v1_1.0_224_frozen.pb
+
+[mobilenet_v1 tensorflow inference accuracy int8]
+ai-type = image_recognition
+model-name = mobilenet_v1
+script = accuracy.sh
+mode = inference
+framework = tensorflow
+device = cpu
+precision = int8
+data-location = /tf_dataset/dataset/TF_Imagenet_FullData
+data-download = To get Imagenet data, please refer to the page "https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md"
+wget = https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mobilenetv1_int8_pretrained_model.pb
diff --git a/notebooks/profiling/profile_utils.py b/notebooks/profiling/profile_utils.py
new file mode 100755
index 000000000..470232a55
--- /dev/null
+++ b/notebooks/profiling/profile_utils.py
@@ -0,0 +1,1467 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2021-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+import sys
+import configparser
+import json
+import os
+import fnmatch
+import psutil
+
+from importlib import util
+tensorflow_found = util.find_spec("tensorflow") is not None
+pytorch_found = util.find_spec("torch") is not None
+pytorch_ext_found = util.find_spec("intel_pytorch_extension") is not None
+
+try:
+    import tensorflow as tf
+    from tensorflow.python.client import timeline
+    from tensorflow.python.training import training_util
+except ImportError as e:
+    print(e)
+    print("can't import tensorflow module")
+    pass
+
+try:
+    from git import Repo
+    has_git = True
+except ImportError as e:
+    print(e)
+    print("can't import git module")
+    has_git = False
+    pass
+
+
+class TimeLiner:
+
+    def __init__(self):
+        self._timeline_dict = None
+
+    def update_timeline(self, chrome_trace):
+        # convert crome trace to python dict
+        chrome_trace_dict = json.loads(chrome_trace)
+        # for first run store full trace
+        if self._timeline_dict is None:
+            self._timeline_dict = chrome_trace_dict
+        # for other - update only time consumption, not definitions
+        else:
+            self._timeline_dict['traceEvents'] += [event for event in chrome_trace_dict['traceEvents'] if 'ts' in event]
+
+    def update_timeline_from_runmeta(self, run_metadata):
+        fetched_timeline = timeline.Timeline(run_metadata.step_stats)
+        chrome_trace = fetched_timeline.generate_chrome_trace_format()
+        self.update_timeline(chrome_trace)
+
+    def save(self, f_name, output_dir=''):
+        import os
+        if output_dir == '':
+            ProfileUtilsRoot = os.environ['ProfileUtilsRoot']
+            output_dir = ProfileUtilsRoot + os.sep + ".."
+        fname_path = output_dir + os.sep + f_name
+        with open(fname_path, 'w') as f:
+            json.dump(self._timeline_dict, f)
+
+
+class tfSession(tf.compat.v1.Session):
+
+    def __init__(self, target='', graph=None, config=None, run_metadata=None, many_runs_timeline=None):
+        self.run_metadata = run_metadata
+        self.options = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
+        self.many_runs_timeline = many_runs_timeline
+        super(tfSession, self).__init__(target, graph=graph, config=config)
+
+    def run(self, c, feed_dict=None, options=None, run_metadata=None):
+        options = self.options
+        run_metadata = self.run_metadata
+        ret = super(tfSession, self).run(c, feed_dict=feed_dict, options=options, run_metadata=run_metadata)
+        self.many_runs_timeline.update_timeline_from_runmeta(run_metadata)
+        return ret
+
+    def save_timeline(self, fname, output_dir=""):
+        fname_path = output_dir + os.sep + fname
+        self.many_runs_timeline.save(fname_path, output_dir=output_dir)
+
+
+class tfProfileHook(tf.estimator.ProfilerHook):
+    def __init__(self, save_steps=None, save_secs=None, output_dir="", json_fname="", timeline_count=10):
+        self._output_tag = "blah-{}"
+        self._output_dir = output_dir
+        self._timer = tf.estimator.SecondOrStepTimer(every_secs=save_secs,
+                                                     every_steps=save_steps)
+        self._atomic_counter = 0
+        self.many_runs_timeline = TimeLiner()
+        self.timeline_count = timeline_count
+
+        self.json_fname = json_fname
+
+    def begin(self):
+        self._next_step = None
+        self._global_step_tensor = training_util.get_global_step()
+
+        if self._global_step_tensor is None:
+            raise RuntimeError("Global step should be created to use ProfilerHook.")
+
+    def before_run(self, run_context):
+        self._request_summary = (self._next_step is None or self._timer.should_trigger_for_step(self._next_step))
+        requests = {}
+        opts = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
+        return tf.estimator.SessionRunArgs(requests, options=opts)
+
+    def after_run(self, run_context, run_values):
+
+        global_step = self._atomic_counter + 1
+        self._atomic_counter = self._atomic_counter + 1
+        self._next_step = global_step + 1
+
+        self.many_runs_timeline.update_timeline_from_runmeta(run_values.run_metadata)
+        if self._atomic_counter == self.timeline_count:
+            self.many_runs_timeline.save(self.json_fname, output_dir=self._output_dir)
+
+    def end(self, session):
+        if self._atomic_counter < self.timeline_count:
+            self.many_runs_timeline.save(self.json_fname, output_dir=self._output_dir)
+
+
+class TensorflowUtils:
+
+    def is_mkl_enabled(self):
+        mkl_enabled = False
+        major_version = int(tf.__version__.split(".")[0])
+        minor_version = int(tf.__version__.split(".")[1])
+        if major_version >= 2:
+            if minor_version < 5:
+                from tensorflow.python import _pywrap_util_port
+            else:
+                from tensorflow.python.util import _pywrap_util_port
+                onednn_enabled = int(os.environ.get('TF_ENABLE_ONEDNN_OPTS', '0'))
+            mkl_enabled = _pywrap_util_port.IsMklEnabled() or (onednn_enabled == 1)
+        else:
+            mkl_enabled = tf.pywrap_tensorflow.IsMklEnabled()
+        return mkl_enabled
+
+
+class GitOps:
+
+    def __init__(self, repopath='./'):
+        self.repopath = repopath
+        if has_git is True:
+            try:
+                self.repo = Repo(repopath)
+            except:
+                print('no repo, init one')
+                self.repo = Repo.init(repopath)
+                self.repo.git.add(A=True)
+                self.repo.git.commit(m='add new files')
+                pass
+        return
+
+    def check_git_status(self):
+        status = self.repo.git.status()
+        if status.find("middle of an am session") != -1 or self.repo.is_dirty():
+            print("middle of an am session or repo is dirty")
+            return False
+        return True
+
+    def find_commit_with_keyword(self, keyword, search_commits_depth=5):
+        ret = False
+        if has_git is False:
+            return ret
+        try:
+            repo = Repo(self.repopath)
+        except:
+            print('no repo, init one')
+            repo = Repo.init(self.repopath)
+            repo.git.add(A=True)
+            repo.git.commit(m='add new files')
+            pass
+
+        if not repo.bare:
+            try:
+                commits = list(repo.iter_commits())[:search_commits_depth]
+                for commit in commits:
+                    if commit.summary.find(keyword) != -1:
+                        ret = True
+                        return ret
+            except Exception:
+                print("EXCEPTION : Find commit %s ", keyword)
+                pass
+        return ret
+
+
+class PlatformUtils:
+
+    def __init__(self):
+        self.cpufreq = ''
+        self.cpu_socket_count = ''
+        self.svmem = ''
+        return
+
+    def dump_platform_info(self):
+        # Import platform_util and print CPU information
+        if 'AIReferenceRoot' in os.environ and os.environ['AIReferenceRoot']:
+            repo_root = os.environ['AIReferenceRoot']
+            platform_util = os.path.join(repo_root, 'benchmarks/common')
+        else:
+            file_dir = os.path.dirname(os.path.abspath(__file__))
+            platform_util = os.path.join(file_dir, '../../benchmarks/common')
+        sys.path.insert(0, platform_util)
+        import platform_util
+        cpu_info = platform_util.CPUInfo()
+        print("=" * 20, "CPU Info", "=" * 20)
+        # number of cores
+        print("Physical cores per socket:", cpu_info.cores_per_socket)
+        print("Total physical cores:", cpu_info.cores)
+        # CPU frequencies
+        cpufreq = psutil.cpu_freq()
+        print("Max Frequency:", cpufreq.max)
+        print("Min Frequency:", cpufreq.min)
+        print("Socket Number:", cpu_info.sockets)
+        print("=" * 20, "Memory Information", "=" * 20)
+        # get the memory details
+        svmem = psutil.virtual_memory()
+        print("Total: ", int(svmem.total / (1024 ** 3)), "GB")
+        self.cpufreq = cpufreq
+        self.cpu_socket_count = cpu_info.sockets
+        self.svmem = svmem
+
+
+class CommonUtils:
+
+    def __init__(self):
+        return
+
+    def found_files_in_folder(self, pattern, path):
+        listOfFiles = os.listdir(path)
+        foundfiles = []
+        founpaths = []
+        for f in listOfFiles:
+            if fnmatch.fnmatch(f, pattern):
+                foundfiles.append(f)
+                founpaths.append(path + os.sep + f)
+        return foundfiles, founpaths
+
+    def found_files_in_folders(self, pattern, paths):
+        foundfiles = []
+        foundpaths = []
+        for path in paths:
+            files, paths = self.found_files_in_folder(pattern, path)
+            foundfiles += files
+            foundpaths += paths
+        return foundfiles, foundpaths
+
+
+class CSVCommonUtils:
+
+    def __init__(self, fnames, filename):
+        self.fnames = fnames
+        self.filename = filename
+        self.create_csv_file()
+        return
+
+    def create_csv_file(self):
+        import csv
+        import os.path
+        if os.path.isfile(self.filename):
+            print('file exists')
+        else:
+            f = open(self.filename, 'w')
+            with f:
+                print("create CSV file with header : ", self.fnames)
+                writer = csv.DictWriter(f, fieldnames=self.fnames)
+                writer.writeheader()
+
+    def write_to_csv(self, dataframes):
+        import csv
+        f = open(self.filename, 'a')
+        with f:
+            writer = csv.DictWriter(f, fieldnames=self.fnames)
+            writer.writerow(
+                {self.fnames[0]: dataframes[0],
+                 self.fnames[1]: dataframes[1]})
+
+    def read_from_csv(self):
+        import csv
+        from collections import defaultdict
+        columns = defaultdict(list)
+
+        if os.path.isfile(self.filename) is False:
+            print('csv file does not exists')
+        f = open(self.filename, 'r')
+        with f:
+            reader = csv.DictReader(f)
+            for row in reader:
+                for (k, v) in row.items():
+                    columns[k].append(v)
+        return columns[self.fnames[0]], columns[self.fnames[1]]
+
+    def delete_csv(self):
+        import os.path
+        import os
+        if os.path.isfile(self.filename):
+            print('file exists')
+            os.remove(self.filename)
+
+class PerfPresenter:
+
+    def __init__(self, showAbsNumber=False):
+        self.tf_util = TensorflowUtils()
+        self.showAbsNumber = showAbsNumber
+        pass
+
+    def autolabel(self, ax, rects):
+        """Attach a text label above each bar in *rects*, displaying its height."""
+        for rect in rects:
+            height = rect.get_height()
+            ax.annotate(
+                '{}'.format(height),
+                xy=(rect.get_x() + rect.get_width() / 2, height),
+                xytext=(0, 3), textcoords='offset points',
+                ha='center', va='bottom')
+
+    def draw_perf_diag(self, topo_name, a_means, b_means, a_label, b_label):
+        import matplotlib.pyplot as plt
+        import numpy as np
+        labels = [topo_name]
+        x = np.arange(len(labels))
+        width = 0.35
+        fig, ax = plt.subplots()
+        rects1 = ax.bar(x - width / 2, a_means, width, label=a_label)
+        rects2 = ax.bar(x + width / 2, b_means, width, label=b_label)
+        ax.set_ylabel('Throughput')
+        ax.set_title('stock TF vs Intel TF')
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels)
+        ax.legend()
+        self.autolabel(ax, rects1)
+        self.autolabel(ax, rects2)
+        fig.tight_layout()
+        plt.show()
+
+    def draw_perf_ratio_diag(self, topo_name, a_means, b_means, a_label, b_label):
+        import matplotlib.pyplot as plt
+        import numpy as np
+        labels = [topo_name]
+        a_ratio = 1
+        b_ratio = float(b_means / a_means)
+        x = np.arange(len(labels))
+        width = 0.35
+        fig, ax = plt.subplots()
+        rects1 = ax.bar(x - width / 2, a_ratio, width, label=a_label)
+        rects2 = ax.bar(x + width / 2, b_ratio, width, label=b_label)
+        ax.set_ylabel('Throughput Speeup ')
+        ax.set_title('stock TF vs Intel TF')
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels)
+        ax.legend()
+        self.autolabel(ax, rects1)
+        self.autolabel(ax, rects2)
+        fig.tight_layout()
+        plt.show()
+
+    def create_csv_logfile(self, Type, filename):
+        import csv
+        import os.path
+        if Type == 'training':
+            fnames = ['mkl', 'elapsed_time']
+        else:
+            fnames = ['mkl', 'elapsed_time', 'throughput', 'accuracy']
+        if os.path.isfile(filename):
+            print('file exists')
+        else:
+            f = open(filename, 'w')
+            with f:
+                writer = csv.DictWriter(f, fieldnames=fnames)
+                writer.writeheader()
+
+    def get_diff_from_csv_filenames(self, x, y):
+        x_split = x.split('_')
+        y_split = y.split('_')
+        if len(x_split) != len(y_split):
+            print("ERROR! can't two files have different formats")
+            return '', ''
+        for i in range(len(x_split)):
+            if x_split[i] != y_split[i]:
+                break
+        return x_split[i], y_split[i]
+
+    def log_infer_perfcsv(self, elapsed_time, throughput, accuracy, filename):
+        import csv
+        f = open(filename, 'a')
+        with f:
+            fnames = ['mkl', 'elapsed_time', 'throughput', 'accuracy']
+            writer = csv.DictWriter(f, fieldnames=fnames)
+            writer.writerow(
+                {'mkl': self.tf_util.is_mkl_enabled(),
+                 'elapsed_time': elapsed_time,
+                 'throughput': throughput,
+                 'accuracy': accuracy})
+
+    def log_train_perfcsv(self, elapsed_time, filename):
+        import csv
+        f = open(filename, 'a')
+        with f:
+            fnames = ['mkl', 'elapsed_time']
+            writer = csv.DictWriter(f, fieldnames=fnames)
+            writer.writerow(
+                {'mkl': self.tf_util.is_mkl_enabled(),
+                 'elapsed_time': elapsed_time})
+
+    def read_number_from_csv(self, filepath, framename):
+        import csv
+        import statistics
+        col1 = []
+        col2 = []
+        mean1 = 0
+        mean2 = 0
+        stdev1 = 0
+        stdev2 = 0
+        if os.path.isfile(filepath) is False:
+            print('csv file does not exists')
+            return (mean1, mean2, len(col1), len(col2), stdev1, stdev2)
+        f = open(filepath, 'r')
+        with f:
+            reader = csv.DictReader(f)
+            for row in reader:
+                if row['mkl'] == 'True':
+                    col1.append(float(row[framename]))
+                else:
+                    col2.append(float(row[framename]))
+
+        if len(col1) > 0:
+            mean1 = statistics.mean(col1)
+        if len(col2) > 0:
+            mean2 = statistics.mean(col2)
+        if len(col1) > 1:
+            stdev1 = statistics.stdev(col1)
+        if len(col2) > 1:
+            stdev2 = statistics.stdev(col2)
+        return (mean1, mean2, len(col1), len(col2), stdev1, stdev2)
+
+    def plot_perf_graph(self, ylabel, xlabel, stock_number, intel_number, stock_stdev, intel_stdev):
+        import matplotlib.pyplot as plt
+        import numpy as np
+        labels = [xlabel]
+        x = np.arange(len(labels))
+        width = 0.35
+        fig, ax = plt.subplots()
+        rects1 = ax.bar(x - width / 2, stock_number, width, yerr=stock_stdev, label='stock TF')
+        rects2 = ax.bar(x + width / 2, intel_number, width, yerr=intel_stdev, label='intel TF')
+        ax.set_ylabel(ylabel)
+        ax.set_title('stock TF vs Intel TF')
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels)
+        ax.legend()
+
+        def autolabel(rects):
+            """Attach a text label above each bar in *rects*, displaying its height."""
+            for rect in rects:
+                height = rect.get_height()
+                ax.annotate(
+                    '{}'.format(height),
+                    xy=(rect.get_x() + rect.get_width() / 2, height),
+                    xytext=(0, 3),
+                    textcoords='offset points',
+                    ha='center',
+                    va='bottom')
+
+        autolabel(rects1)
+        autolabel(rects2)
+        fig.tight_layout()
+        plt.show()
+
+    def plot_perf_graph_v2(self, ylabel, xlabel, means_list, stddev_list, filepath_list, label_list, title='stock TF vs Intel TF'):
+        import matplotlib.pyplot as plt
+        import numpy as np
+        labels = [xlabel]
+        x = np.arange(len(labels))
+        width = 0.35
+        fig, ax = plt.subplots()
+
+        rects_list = []
+        for i in range(len(filepath_list)):
+            number = means_list[i]
+            stddev = stddev_list[i]
+            label = label_list[i]
+            rects = ax.bar(x - width / 2 + width * i, number, width, yerr=stddev, label=label)
+            rects_list.append(rects)
+
+        ax.set_ylabel(ylabel)
+        ax.set_title(title)
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels)
+        ax.legend()
+
+        def autolabel(rects):
+            """Attach a text label above each bar in *rects*, displaying its height."""
+            for rect in rects:
+                height = rect.get_height()
+                ax.annotate(
+                    '{}'.format(height),
+                    xy=(rect.get_x() + rect.get_width() / 2, height),
+                    xytext=(0, 3),
+                    textcoords='offset points',
+                    ha='center',
+                    va='bottom')
+
+        for i in range(len(rects_list)):
+            autolabel(rects_list[i])
+        fig.tight_layout()
+        plt.show()
+
+    def draw_perf_diag_from_csv(self, filepath, framename, y_axis_name, title):
+        if self.showAbsNumber is False:
+            return
+        stock_means = []
+        intel_means = []
+        stock_stdev = []
+        intel_stdev = []
+        mean1, mean2, len1, len2, stdev1, stdev2 = self.read_number_from_csv(filepath, framename)
+        if mean1 == 0 or mean2 == 0:
+            print("ERROR. Users must run the benchmark with both Stock TF and Intel TF\n")
+            return
+        stock_means.append(float(mean2))
+        intel_means.append(float(mean1))
+        stock_stdev.append(float(stdev2))
+        intel_stdev.append(float(stdev1))
+        self.plot_perf_graph(y_axis_name, title, stock_means, intel_means, stock_stdev, intel_stdev)
+
+    def draw_perf_ratio_diag_from_csv(self, filepath, framename, y_axis_name, title):
+        stock_ratio_means = [1]
+        intel_ratio_means = []
+        stock_stdev = []
+        intel_stdev = []
+        mean1, mean2, len1, len2, stdev1, stdev2 = self.read_number_from_csv(filepath, framename)
+        if mean1 == 0 or mean2 == 0:
+            print("ERROR. Users must run the benchmark with both Stock TF and Intel TF\n")
+            return
+        intel_ratio_means.append(float(mean1 / mean2))
+        stock_stdev.append(float(stdev2 / mean2))
+        intel_stdev.append(float(stdev1 / mean1))
+        self.plot_perf_graph(y_axis_name, title, stock_ratio_means, intel_ratio_means, stock_stdev, intel_stdev)
+
+    def draw_perf_diag_from_csvs(self, filepath_list, label_list, framename, y_axis_name, x_axis_name, title, analyze_mkl=True):
+        if self.showAbsNumber is False:
+            return
+        means_list = []
+        stddev_list = []
+        for filepath in filepath_list:
+            means = []
+            stdev = []
+            mean1, mean2, len1, len2, stdev1, stdev2 = self.read_number_from_csv(filepath, framename)
+            if analyze_mkl is True:
+                if mean1 == 0:
+                    print("ERROR. No perf number from CSV file\n")
+                    return
+                means.append(float(mean1))
+                stdev.append(float(stdev1))
+            else:
+                if mean2 == 0:
+                    print("ERROR. No perf number from CSV file\n")
+                    return
+                means.append(float(mean2))
+                stdev.append(float(stdev2))
+            means_list.append(means)
+            stddev_list.append(stdev)
+        self.plot_perf_graph_v2(y_axis_name, x_axis_name, means_list, stddev_list, filepath_list, label_list, title=title)
+
+    def draw_perf_ratio_diag_from_csvs(self, filepath_list, label_list, framename, y_axis_name, x_axis_name, title, analyze_mkl=True):
+        means_list = []
+        stddev_list = []
+        dividend = 0
+        for filepath in filepath_list:
+            means = []
+            stdev = []
+            ratio_means = []
+            mean1, mean2, len1, len2, stdev1, stdev2 = self.read_number_from_csv(filepath, framename)
+            if analyze_mkl is True:
+                if mean1 == 0:
+                    print("ERROR. No perf number from CSV file\n")
+                    return
+                means.append(float(mean1))
+                stdev.append(0)
+                if dividend == 0:
+                    dividend = mean1
+                ratio_means.append(float(mean1 / dividend))
+            else:
+                if mean2 == 0:
+                    print("ERROR. No perf number from CSV file\n")
+                    return
+                means.append(float(mean2))
+                stdev.append(0)
+                if dividend == 0:
+                    dividend = mean2
+                ratio_means.append(float(mean2 / dividend))
+            means_list.append(ratio_means)
+            stddev_list.append(stdev)
+        self.plot_perf_graph_v2(y_axis_name, x_axis_name, means_list, stddev_list, filepath_list, label_list, title=title)
+
+    def parse_stdout(self, output, keyword):
+        output = str(output)
+        output = output.split("\n")
+        lines = output[0].split('\\')
+        for l in lines:
+            if l.find("Throughput") > 0:
+                return l
+        return None
+
+    def read_throughput(self, filepath, keyword='Throughput', index=1, splitter=' '):
+        number = None
+        with open(filepath) as (fp):
+            line = fp.readline()
+            while line:
+                line = fp.readline()
+                if line.find(keyword) != -1:
+                    number = line.split(splitter)[index]
+        return number
+
+    def read_accuracy(self, filepath, keyword='accuracy', index=-2):
+        accuracy = []
+        with open(filepath) as (fp):
+            line = fp.readline()
+            while line:
+                line = fp.readline()
+                if line.find(keyword) != -1:
+                    number = line.split(' ')[index]
+                    number = number.strip(',')
+                    number = number.strip('(')
+                    number = number.strip(')')
+                    accuracy.append(float(number))
+        return accuracy
+
+    def read_iteration_time(self, filepath, keyword='Iteration', index=-2):
+        iteration = []
+        with open(filepath) as (fp):
+            line = fp.readline()
+            while line:
+                line = fp.readline()
+                if line.find(keyword) != -1:
+                    number = line.split(' ')[index]
+                    number = number.strip(',')
+                    number = number.strip('(')
+                    number = number.strip(')')
+                    iteration.append(float(number))
+        return iteration
+
+
+class AIReferenceConfigFile:
+
+    def __init__(self, confpath='profiling/ai_reference_models.ini'):
+        self.configpath = confpath
+        self.wget = ''
+        self.data_download = ''
+        self.data_location = ''
+        if tensorflow_found == True:
+            self.tf_util = TensorflowUtils()
+        self.model_name = ''
+        self.script = ''
+        self.mode = ''
+        self.framework = ''
+        self.device = ''
+        self.precision = ''
+        self.ai_type = ''
+        self.custom_args = ''
+        self.json_fname = ''
+        self.json_fname = 'stock_'
+        self.patches = ''
+        self.patched = False
+        self.patches_keyword = ''
+
+    def read_section(self):
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        return config.sections()
+
+    def read_supported_section(self):
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        supported_sections = []
+        for each_section in config.sections():
+            is_supported = True
+            if is_supported is True:
+                supported_sections.append(each_section)
+
+        return supported_sections
+
+    def convert_configs_to_pd_dataframe(self):
+
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        import pandas as pd
+
+        index_list = []
+        data_list = []
+        columns_list = ['benchmark', 'model-name', 'mode', 'precision', 'device', 'framework', 'patches', 'json-fname']
+        for section in config.sections():
+            index_list.append(section)
+            data = []
+            data.append(section)
+            data.append(config.get(section, columns_list[1]))
+            data.append(config.get(section, columns_list[2]))
+            data.append(config.get(section, columns_list[3]))
+            data.append(config.get(section, columns_list[4]))
+            data.append(config.get(section, columns_list[5]))
+            data.append(config.get(section, columns_list[6]))
+            data.append(config.get(section, columns_list[7]))
+            data_list.append(data)
+        df = pd.DataFrame(data_list, columns=columns_list)
+
+        df_types = df.groupby([columns_list[1], columns_list[2]]).filter(lambda x: len(x) >= 2)
+
+        df_types_obj = df_types.groupby([columns_list[1], columns_list[2]])
+
+        return df, df_types, df_types_obj
+
+    def read_value_from_section(self, model_name, key):
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        string_val = config.get(model_name, key)
+        return string_val
+
+    def write_value_from_section(self, model_name, key, val):
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        config.set(model_name, key, val)
+
+        # save to a file
+        with open(self.configpath, 'w') as configfile:
+            config.write(configfile)
+        return
+
+    def parsing_custom_args(self, model_name, custom_args):
+        configvals = []
+        if custom_args.count('$') >= 2:
+            read_var = custom_args.split('$')[1]
+            replace_var = self.read_value_from_section(model_name, read_var)
+            custom_args = custom_args.replace('$' + read_var + '$', replace_var)
+        custom_args_list = custom_args.split(" ")
+        for arg in custom_args_list:
+            if arg != '':
+                configvals.append(arg)
+        return configvals
+
+    def read_config(self, model_name):
+        configs = []
+        config = configparser.ConfigParser()
+        config.read(self.configpath)
+        for each_section in config.sections():
+            if each_section == model_name:
+                for each_key, each_val in config.items(each_section):
+                    key = '--' + each_key
+                    if each_key == 'wget':
+                        if each_val is not None:
+                            self.wget = each_val
+                    elif each_key == 'data-download':
+                        if each_val is not None:
+                            self.data_download = each_val
+                    elif each_key == 'data-location':
+                        if each_val is not None:
+                            if each_val != '':
+                                configs.append(key)
+                                configs.append(each_val)
+                                self.data_location = each_val
+                    elif each_key == 'batch-size':
+                        if len(each_val) != 0:
+                            self.batch_size = int(each_val)
+                    elif each_key == 'model-name':
+                        if each_val is not None:
+                            self.model_name = each_val
+                    elif each_key == 'ai-type':
+                        if each_val is not None:
+                            self.ai_type = each_val
+                    elif each_key == 'script':
+                        if each_val is not None:
+                            self.script = each_val
+                    elif each_key == 'mode':
+                        if each_val is not None:
+                            self.mode = each_val
+                    elif each_key == 'framework':
+                        if each_val is not None:
+                            self.framework = each_val
+                    elif each_key == 'device':
+                        if each_val is not None:
+                            self.device = each_val
+                    elif each_key == 'precision':
+                        if each_val is not None:
+                            self.precision = each_val
+                    elif each_key == 'custom-args':
+                        if len(each_val) != 0:
+                            self.custom_args = each_val
+                    elif each_key == 'json-fname':
+                        if each_val is not None:
+                            self.json_fname = self.json_fname + each_val
+                    elif each_key == 'patches':
+                        if each_val is not None:
+                            self.patches = each_val
+                            val = each_val.split('.')[0]
+                            val = val.split('-')[1:]
+                            keyword = " ".join(val)
+                            self.patches_keyword = keyword
+                    else:
+                        if len(each_val) != 0:
+                            if each_val[0] == '=':
+                                configs.append(key + each_val)
+                            else:
+                                configs.append(key)
+                                configs.append(each_val)
+
+        return configs
+
+    def get_parameters(
+            self, model_name, configvals):
+        benchmark_argvs = []
+        benchmark_argvs = benchmark_argvs + configvals
+        return benchmark_argvs
+
+    def find_pretrained_model_in_folder(self, uncompress_folderpath, pattern='*.pb'):
+        utils = CommonUtils()
+        if uncompress_folderpath.find(pattern[1:]) != -1:
+            return uncompress_folderpath
+        pb_filename = ''
+        pb_files, pb_paths = utils.found_files_in_folder(pattern, uncompress_folderpath)
+        if len(pb_files) != 0:
+            pb_filename = os.sep + pb_files[0]
+        pretrain_model_path = uncompress_folderpath + pb_filename
+        return pretrain_model_path
+
+    def untar_file(self, filepath, extracted_fd='./'):
+        import tarfile
+        extractpath = ''
+        tar = tarfile.open(filepath)
+        for tarinfo in tar:
+            if tarinfo.isdir():
+                print(tarinfo.name)
+                extractpath = tarinfo.name
+                break
+        target_path = extracted_fd + os.sep + extractpath
+        if os.path.exists(target_path) is False:
+            tar.extractall(path=extracted_fd)
+        tar.close()
+        return extractpath
+
+    def uncompress_file(self, filepath, pretrainfd='pretrained', current_path='./'):
+        import shutil
+        uncompress_path = filepath
+        full_filename = filepath.split(os.sep)[-1]
+
+        file_ext = full_filename.split('.')[-1]
+        filename = full_filename.split('.')[0]
+        cmd = ''
+        if file_ext == 'zip':
+            cmd = "unzip " + filepath
+        elif file_ext == 'gz':
+            uncompress_foldername = self.untar_file(filepath, extracted_fd=pretrainfd)
+            uncompress_path = filepath.replace(full_filename, uncompress_foldername)
+        if cmd != '':
+            os.system(cmd)
+            if os.path.exists(pretrainfd + os.sep + filename) is False:
+                shutil.move(filename, pretrainfd)
+
+            uncompress_path = os.path.splitext(filepath)[0]
+
+        return uncompress_path
+
+    def download_dataset(self, datasetfd='dataset', current_path='./'):
+        cmd = self.data_download
+        dataset_path = current_path + os.sep + datasetfd
+        if os.path.exists(dataset_path) is True:
+            return dataset_path
+        os.system(cmd)
+        print('Downloaded the model in:', dataset_path)
+        return dataset_path
+
+    def download_pretrained_model(self, pretrainfd='pretrained', current_path='./'):
+        import shutil
+        cmd = "wget " + self.wget + ' -P ' + current_path + os.sep + pretrainfd
+        filename = self.wget.split('/')[-1]
+        pretrain_model_path = current_path + os.sep + pretrainfd + os.sep + filename
+        if os.path.exists(pretrain_model_path) is True:
+            return pretrain_model_path
+        os.system(cmd)
+        if os.path.exists(pretrainfd) is False:
+            os.mkdir(pretrainfd)
+        if os.path.exists(pretrainfd + os.sep + filename) is False:
+            shutil.move(filename, pretrainfd)
+        print('Downloaded the model in:', pretrain_model_path)
+        return pretrain_model_path
+    
+    def patch_model_to_enable_timeline(self, patch_fd="profiling/patches", repopath=""):
+        if self.patches == '':
+            return False
+        self.git = GitOps(repopath=repopath)
+        print("patch keyword: ", self.patches_keyword)
+        if self.git.find_commit_with_keyword(self.patches_keyword) is True:
+            self.patched = True
+            print("has been patched, no action")
+            return True
+        print(self.patches)
+        patch_path = patch_fd + os.sep + self.patches
+        if os.path.exists(patch_path) is True:
+            cmd = "git am " + patch_path
+            print(cmd)
+            os.system(cmd)
+            self.patched = True
+        else:
+            print("couldn't find the patch file : ", patch_path)
+        ret = self.git.check_git_status()
+        return ret
+
+    def unpatch_model_to_enable_timeline(self, model_path='../models/'):
+        if self.patched is False:
+            print("has not been patched, no action")
+            return True
+        print("unpatch keyword: ", self.patches_keyword)
+        if self.git.find_commit_with_keyword(self.patches_keyword, search_commits_depth=2) is True:
+            import os
+            print("do unpatch")
+            cmd = "git reset HEAD^ "
+            print(cmd)
+            os.system(cmd)
+            cmd = "git checkout " + model_path + "*"
+            print(cmd)
+            os.system(cmd)
+        ret = self.git.check_git_status()
+        return ret
+
+class TFTimelinePresenter:
+
+    def __init__(self, showAbsNumber=False):
+        self.showAbsNumber = showAbsNumber
+        import pandas as pd
+        pd.set_option('display.max_rows', 500)
+        pd.set_option('display.max_columns', 500)
+        pd.set_option('display.width', 1500)
+        self.pd = pd
+
+    def show(self, diag, Type):
+        import matplotlib.pyplot as plt
+        ret = None
+        if self.showAbsNumber is True or Type == 'pie':
+            ret = diag
+            plt.show()
+        plt.clf()
+        return ret
+
+    def read_timeline(self, fn, maxents=0):
+        import json
+        import pandas as pd
+        with open(fn, 'r') as (f):
+            j = json.loads(f.read())
+        allkeys = [list(js.keys()) for js in j['traceEvents']]
+        allkeys = [item for sublist in allkeys for item in iter(sublist)]
+        allkeys = sorted(set(allkeys))
+        argkeys = [js['args'].keys() for js in j['traceEvents'] if 'args' in js]
+        argkeys = sorted(set([item for sublist in argkeys for item in iter(sublist)]))
+        argkeys = ['arg_' + k for k in argkeys]
+        entries = []
+        for i, e in enumerate(j['traceEvents']):
+            if maxents != 0 and i > maxents:
+                break
+            ent = {}
+            for k, v in e.items():
+                if k == 'args':
+                    for a in v.keys():
+                        ent['arg_' + a] = str(v[a])
+
+                else:
+                    ent[k] = str(v)
+
+            entries.append(ent)
+
+        df = pd.DataFrame(entries)
+        return df
+
+    def summarize_item(self, tl, item, ascending=False):
+        return tl.groupby([item])['dur'].sum().sort_values(ascending=ascending)
+
+    def summarize_barh(self, tl, item, topk=15, ascending=False, ax=None, title=None, figsize=None, logx=False):
+        ret = self.summarize_item(tl, item, ascending)[:topk].plot.barh(
+            ax=ax, title=title, figsize=figsize, logx=False)
+        return ret
+
+    def summarize_pie(self, tl, item, topk=15, ascending=False, ax=None, title=None, figsize=None, logx=False):
+        ret = self.summarize_item(tl, item, ascending)[:topk].plot.pie(
+            ax=ax, title=title, figsize=figsize, logx=logx, autopct='%1.1f%%')
+        return ret
+
+    def opname(self, x):
+        return self.demangle(x.split('/')[(-1)])
+
+    def opbase(self, x):
+        return self.opname(x).split(':')[(-1)]
+
+    def blockname(self, x):
+        return x.split('/')[0].split('_')[0]
+
+    def postprocess_timeline(self, t):
+        t['dur'] = t['dur'].astype('float') / 1000
+        t['ts'] = t['ts'].astype('float') - t['ts'].astype('float').min()
+        t['arg_name'] = t['arg_name'].astype('str').replace('nan', '')
+        t['arg_opname'] = t['arg_name'].apply(self.opname)
+        t['arg_opbase'] = t['arg_name'].apply(self.opbase)
+        return t
+
+    def demangle(self, x, short=True):
+        import cxxfilt
+        z = x.split('::')
+        try:
+            if short:
+                return z[0] + '::' + cxxfilt.demangle(z[1]).split('<')[0]
+            else:
+                return z[0] + '::' + cxxfilt.demangle(z[1])
+        except IndexError:
+            return x
+
+    def get_tf_ops_time(self, timeline_pd, fn, tfile_prefix):
+        sitems = self.summarize_item(timeline_pd, 'arg_op')
+        import csv
+        filename = fn.split('.')[0] + '.csv'
+        f = open(filename, 'w')
+        time_col_name = 'elapsed_time_' + tfile_prefix
+        total_time = 0.0
+        total_mkl_time = 0.0
+        with f:
+            fnames = ['op', time_col_name, 'speedup', 'mkl_op', 'native_op']
+            writer = csv.DictWriter(f, fieldnames=fnames)
+            writer.writeheader()
+            x = 0
+            for sitem in sitems:
+                mkl_op = False
+                native_op = False
+                if tfile_prefix == 'mkl':
+                    op_name = sitems.index[x].strip('_')
+                    if op_name.find('Mkl') != -1:
+                        mkl_op = True
+                        op_name = op_name[3:]
+                        total_mkl_time += float(sitems[x])
+                        if op_name.find('Native') != -1:
+                            op_name = op_name[6:]
+                            native_op = True
+                    total_time += float(sitems[x])
+                else:
+                    op_name = sitems.index[x].strip('_')
+                writer.writerow({'op': op_name, time_col_name: sitems[x], 'speedup': 0, 'mkl_op': mkl_op, 'native_op': native_op})
+                x = x + 1
+
+        percentage_filename = ''
+        if total_mkl_time > 0.0:
+            percentage_filename = fn.split('.')[0] +'_mkl_percentage' +'.csv'
+            f = open(percentage_filename, 'w')
+            with f:
+                fnames = ['op', 'time']
+                writer = csv.DictWriter(f, fieldnames=fnames)
+                writer.writeheader()
+                writer.writerow({'op': 'mkl_op', 'time': total_mkl_time})
+                writer.writerow({'op': 'nono_mkl_op', 'time': total_time - total_mkl_time})
+
+        ret = None, percentage_filename
+        if self.showAbsNumber is True:
+            ret = sitems, percentage_filename
+        return ret
+
+    def plot_summary_barth(self, timeline_pd, tfile_prefix):
+        filename = tfile_prefix + '_tf_op_duration_bar.png'
+        title_ = tfile_prefix + 'TF : op duration bar chart'
+        ax = self.summarize_barh(timeline_pd, 'arg_op', title=title_, topk=50, logx=True, figsize=(10,
+                                                                                                   10))
+        ax.figure.savefig(filename, bbox_inches='tight')
+
+    def plot_summary_pie(self, timeline_pd, tfile_prefix):
+        filename = tfile_prefix + '_tf_op_duration_pie.png'
+        title_ = tfile_prefix + 'TF : op duration pie chart'
+        timeline_pd_known = timeline_pd[(~timeline_pd['arg_op'].str.contains('unknown'))]
+        ax = self.summarize_pie(timeline_pd_known, 'arg_op', title=title_, topk=50, logx=True, figsize=(10,
+                                                                                                        10))
+        ax.figure.savefig(filename, bbox_inches='tight')
+
+    def merge_two_csv_files(self, merged_filepath, a, b):
+        merged = a.merge(b, on='op')
+        merged['speedup'] = merged['elapsed_time_stock'] / merged['elapsed_time_mkl']
+        if merged['mkl_op_x'] is True:
+            merged['mkl_op'] = True
+        merged['mkl_op'] = merged['mkl_op_x'] + merged['mkl_op_y']
+        if merged['native_op_x'] is True:
+            merged['native_op'] = True
+        merged['native_op'] = merged['native_op_x'] + merged['native_op_y']
+        merged = merged.drop(columns=['speedup_x', 'speedup_y'])
+        merged = merged.drop(columns=['mkl_op_x', 'mkl_op_y'])
+        merged = merged.drop(columns=['native_op_x', 'native_op_y'])
+        if self.showAbsNumber is False:
+            ret = merged.drop(columns=['elapsed_time_stock', 'elapsed_time_mkl'])
+        else:
+            ret = merged
+        merged.to_csv(merged_filepath, index=False)
+        return ret
+
+    def create_csv_among_extra_common_ops(self, extra, common, fpath, tag1, tag2):
+        import pandas as pd
+        extra = extra.rename(columns={extra.columns.values[1]: tag1})
+        extra = extra.drop(columns=['speedup'])
+        common_time = common[common.columns.values[1]].sum(axis=0)
+        append_op = 'Common ops with ' + tag2
+        to_append = [append_op, common_time, True, False]
+        series = pd.Series(to_append, index=extra.columns)
+        extra = extra.append(series, ignore_index=True)
+        extra.to_csv(fpath, index=False)
+        return extra
+
+    def merge_two_csv_files_v2(self, merged_filepaths, a, b, tags=['stock', 'intel']):
+        merged = a.merge(b, on='op')
+        extra_a = a[~a.op.isin(merged.op)]
+        common_a = a[a.op.isin(merged.op)]
+        extra_b = b[~b.op.isin(merged.op)]
+        common_b = b[b.op.isin(merged.op)]
+        merged['speedup'] = merged.iloc[:, 1] / merged.iloc[:, 5]
+        merged = merged.rename(columns={merged.columns.values[1]: tags[0], merged.columns.values[5]: tags[1]})
+        if merged.iloc[:, 3] is True:
+            merged['mkl_op'] = True
+        merged['mkl_op'] = merged.iloc[:, 3] + merged.iloc[:, 7]
+        if merged.iloc[:, 4] is True:
+            merged['native_op'] = True
+        merged['native_op'] = merged.iloc[:, 4] + merged.iloc[:, 8]
+        merged = merged.drop(columns=[merged.columns.values[2], merged.columns.values[3], merged.columns.values[4], merged.columns.values[6], merged.columns.values[7], merged.columns.values[8]])
+        if self.showAbsNumber is False:
+            ret = merged.drop(columns=[merged.columns.values[1], merged.columns.values[5]])
+        else:
+            ret = merged
+        merged.to_csv(merged_filepaths[0], index=False)
+
+        fpath = merged_filepaths[1]
+        self.create_csv_among_extra_common_ops(extra_a, common_a, fpath, tags[0], tags[1])
+
+        fpath = merged_filepaths[2]
+        self.create_csv_among_extra_common_ops(extra_b, common_b, fpath, tags[1], tags[0])
+
+        return ret
+
+    def compare_bar_pie_charts(self, chart_type):
+        import matplotlib.pyplot as plt
+        import matplotlib.image as img
+        from matplotlib import rcParams
+        import os
+        if chart_type == 'bar':
+            imgfiles = [x for x in os.listdir('.') if '_tf_op_duration_bar.png' == x[-23:]]
+        elif chart_type == 'pie':
+            imgfiles = [x for x in os.listdir('.') if '_tf_op_duration_pie.png' == x[-23:]]
+        else:
+            return
+        rcParams['figure.figsize'] = (30, 30)
+        fig, ax = plt.subplots(1, 2)
+        index = 0
+        for imgf in imgfiles:
+            image = img.imread(imgf)
+            ax[index].imshow(image)
+            index = index + 1
+
+    def plot_compare_bar_charts(self, fpath, tags=['stock', 'intel'], num_hotspots=20):
+        if self.showAbsNumber is False:
+            return
+        import numpy as np
+        import matplotlib.pyplot as plt
+        import csv
+        reader = csv.DictReader(open(fpath))
+        xlabels = []
+        a_means = []
+        b_means = []
+        item_name = reader.fieldnames[0]
+        a_name = reader.fieldnames[1]
+        b_name = reader.fieldnames[2]
+        index = 0
+        for row in reader:
+            if row['op'] != 'unknown' and index < num_hotspots:
+                xlabels.append(row[item_name] + "_(mkl-" + str(row['mkl_op']) + ')')
+                b_means.append(float(row[b_name]))
+                a_means.append(float(row[a_name]))
+                index = index + 1
+
+        N = len(xlabels)
+        ind = np.arange(N)
+        width = 0.35
+        fig = plt.figure(figsize=(18, 15))
+        ax = fig.add_subplot(111)
+        rects1 = ax.bar(ind, a_means, width, color='orange')
+        rects2 = ax.bar(ind + width, b_means, width, color='royalblue')
+        ax.set_ylabel('Elpased Time (ms)', fontsize=20)
+        ax.set_title('TF Ops Performance Comparison ', fontdict={'fontsize': 28})
+        ax.set_xticks(ind + width / 2)
+        ax.set_xticklabels(xlabels, rotation=45, rotation_mode="anchor")
+        ax.legend((rects1[0], rects2[0]), [tags[0], tags[1]], fontsize=20)
+        filename = 'compared_tf_op_duration_bar.png'
+        plt.savefig(filename, bbox_inches='tight', pad_inches=0.1)
+        plt.show()
+
+    def plot_compare_ratio_bar_charts(self, fpath, tags=['stock', 'intel'], num_hotspots=20, max_speedup=100):
+        import numpy as np
+        import matplotlib.pyplot as plt
+        import csv
+        reader = csv.DictReader(open(fpath))
+        xlabels = []
+        b_xlabels = []
+        c_xlabels = []
+        a_means = []
+        b_means = []
+        c_means = []
+        item_name = reader.fieldnames[0]
+        c_name = reader.fieldnames[3]
+        index = 0
+        for row in reader:
+            if row['op'] != 'unknown' and index < num_hotspots:
+
+                if float(row[c_name]) > max_speedup:
+                    speedup_val = max_speedup
+                else:
+                    speedup_val = float(row[c_name])
+
+                if str(row['mkl_op']) == 'True':
+                    b_xlabels.append(row[item_name])
+                    b_means.append(speedup_val)
+                    a_means.append(1)
+                else:
+                    c_xlabels.append(row[item_name])
+                    c_means.append(speedup_val)
+                    a_means.append(1)
+                index = index + 1
+        xlabels = b_xlabels + c_xlabels
+        b_N = len(b_xlabels)
+        N = len(xlabels)
+        b_ind = np.arange(b_N)
+        ind = np.arange(N)
+        c_ind = ind[b_N:]
+        width = 0.35
+        fig = plt.figure(figsize=(18, 15))
+        ax = fig.add_subplot(111)
+        rects2 = ax.bar(b_ind + width / 2, b_means, width, color='royalblue')
+        rects3 = ax.bar(c_ind + width / 2, c_means, width, color='orange')
+        rects = rects2 + rects3
+        ax.set_ylabel('Speedup', fontsize=20)
+        ax.set_title('TF Ops Performance Comparison ', fontdict={'fontsize': 28})
+        ax.set_xticks(ind + width / 2)
+        ax.set_xticklabels(xlabels, rotation=45, rotation_mode="anchor")
+        ax.legend([rects[0]], [tags[1]], fontsize=20)
+        plt.axhline(y=1, linewidth=4, color='r')
+        filename = 'compared_tf_op_duration_ratio_bar.png'
+        plt.savefig(filename, bbox_inches='tight', pad_inches=0.1)
+        plt.show()
+
+    def plot_pie_chart(self, fpath, tag):
+        import matplotlib.pyplot as plt
+        import csv
+        import matplotlib as mpl
+        mpl.rcParams['font.size'] = 9.0
+        reader = csv.DictReader(open(fpath))
+        xlabels = []
+        a_means = []
+        item_name = reader.fieldnames[0]
+        a_name = reader.fieldnames[1]
+
+        for row in reader:
+            if row['op'] != 'unknown':
+                xlabels.append(row[item_name])
+                a_means.append(float(row[a_name]))
+
+        fig = plt.figure(figsize=(18, 15))
+
+        ax1 = fig.add_axes([0, 0, .5, .5], aspect=1)
+        wedges, texts, autotexts = ax1.pie(
+            a_means, autopct='%1.1f%%',
+            textprops=dict(color="w", fontsize=18), radius=1.2)
+        ax1.set_title(tag, fontdict={'fontsize': 28})
+
+        box = ax1.legend(
+            wedges,
+            xlabels,
+            title="TF Ops",
+            loc="center left",
+            bbox_to_anchor=(1, 0, 0.5, 1),
+            fontsize=20)
+        box.get_title().set_fontsize(20)
+        filename = 'tf_op_duration_pie.png'
+        plt.savefig(filename, bbox_inches='tight', pad_inches=0.1)
+        plt.show()
+
+    def plot_compare_pie_charts(self, fpath, tags=['stock', 'intel']):
+        import matplotlib.pyplot as plt
+        import csv
+        import matplotlib as mpl
+        mpl.rcParams['font.size'] = 9.0
+        reader = csv.DictReader(open(fpath))
+        xlabels = []
+        a_means = []
+        b_means = []
+        item_name = reader.fieldnames[0]
+        a_name = reader.fieldnames[1]
+        b_name = reader.fieldnames[2]
+
+        for row in reader:
+            if row['op'] != 'unknown':
+                xlabels.append(row[item_name])
+                b_means.append(float(row[b_name]))
+                a_means.append(float(row[a_name]))
+
+        fig = plt.figure(figsize=(18, 15))
+
+        ax1 = fig.add_axes([0, 0, .5, .5], aspect=1)
+        wedges, texts, autotexts = ax1.pie(
+            a_means, autopct='%1.1f%%',
+            textprops=dict(color="w", fontsize=18), radius=1.2)
+        ax1.set_title(tags[0], fontdict={'fontsize': 28})
+
+        ax2 = fig.add_axes([.5, .0, .5, .5], aspect=1)
+        wedges, texts, autotexts = ax2.pie(
+            b_means, autopct='%1.1f%%',
+            textprops=dict(color="w", fontsize=18), radius=1.2)
+        ax2.set_title(tags[1], fontdict={'fontsize': 28})
+
+        box = ax2.legend(
+            wedges,
+            xlabels,
+            title="TF Ops",
+            loc="center left",
+            bbox_to_anchor=(1, 0, 0.5, 1),
+            fontsize=20)
+        box.get_title().set_fontsize(20)
+        filename = 'compared_tf_op_duration_pie.png'
+        plt.savefig(filename, bbox_inches='tight', pad_inches=0.1)
+        plt.show()
+
+
+class oneDNNLog:
+
+    def __init_(self):
+        self.filename = ''
+        self.data = None
+        self.exec_data = None
+        return
+
+    def load_log(self, log):
+        self.filename = log
+
+        data = self.load_log_dnnl(log)
+        count = data['time'].count()
+
+        if count == 0:
+            data = self.load_log_mkldnn(log)
+            count = data['time'].count()
+
+        exec_data = data[data['exec'] == 'exec']
+        self.data = data
+        self.exec_data = exec_data
+        return
+
+    def load_log_dnnl(self, log):
+        import pandas as pd
+        # dnnl_verbose,exec,cpu,convolution,jit:avx2,forward_inference,src_f32::blocked:abcd:f0 wei_f32::blocked:Acdb8a:f0 bia_f32::blocked:a:f0 dst_f32::blocked:aBcd8b:f0,,alg:convolution_direct,mb1_ic3oc96_ih227oh55kh11sh4dh0ph0_iw227ow55kw11sw4dw0pw0,1.21704
+        data = pd.read_csv(log, names=['dnnl_verbose', 'exec', 'arch', 'type', 'jit', 'pass', 'fmt', 'opt', 'alg', 'shape', 'time'])
+        return data
+
+    def load_log_mkldnn(self, log):
+        import pandas as pd
+        # mkldnn_verbose,exec,convolution,jit:avx512_common,forward_training,fsrc:nChw16c fwei:OIhw16i16o fbia:undef fdst:nChw16c,alg:convolution_direct,mb100_ic128oc32_ih7oh7kh3sh1dh0ph1_iw7ow7kw3sw1dw0pw1,0.201904
+        print("load_log_mkldnn")
+        data = pd.read_csv(log, names=['mkldnn_verbose', 'exec', 'type', 'jit', 'pass', 'fmt', 'alg', 'shape', 'time'])
+        return data
+
+
+class oneDNNUtils:
+
+    def __init_(self):
+        self.topk = 50
+        self.logx = True
+        self.figsize = (10, 10)
+        import matplotlib.pyplot as plt
+        fig = plt.figure(figsize=(18, 15))
+        self.ax = fig.add_subplot(111)
+        return
+
+    def breakdown(self, data, Group, Type):
+        import matplotlib.pyplot as plt
+        import numpy as np
+        fig = plt.figure(figsize=(18, 15))
+        ax = fig.add_subplot(111)
+        figsize = (10, 10)
+        topk = 50
+        if Type == "time":
+            print()
+            print(' breakdown:', Group)
+            if data['time'].dtypes != np.dtype('float64'):
+                data['time'] = data['time'].astype(float)
+            time = data.groupby(Group)['time'].sum().sort_values().head(topk)
+            print(time)
+            title = Group + "Time Breakdown"
+            time[:topk].plot.pie(
+                ax=ax, title=title, figsize=figsize, logx=True, textprops=dict(fontsize=18), autopct='%1.1f%%')
+            ax.figure.savefig(title)
+        elif Type == "count":
+            print()
+            count = data[Group].value_counts().head(topk)
+            print(count)
+            title = Group + "Count Breakdown"
+            count[:topk].plot.bar(
+                ax=ax, title=title, figsize=figsize, logx=False, rot=45)
+            ax.figure.savefig(title)
+        return
+
+    def stats_comp(self, name, Type, onednn_log1, onednn_log2, n=50, tags=['run1', 'run2']):
+        import pandas as pd
+        import matplotlib.pyplot as plt
+        import numpy as np
+        fig = plt.figure(figsize=(18, 15))
+        ax = fig.add_subplot(111)
+        figsize = (10, 10)
+        topk = 50
+        column_name = tags[1] + '/' + tags[0]
+        d1 = onednn_log1.exec_data
+        log1 = onednn_log1.filename
+        d2 = onednn_log2.exec_data
+        log2 = onednn_log2.filename
+        print(name, 'stats:')
+        if Type == "count":
+            jitstat = pd.concat((d1[name].value_counts(), d2[name].value_counts()), axis=1, sort=True)
+            jitstat.columns = ('1-' + log1, '2-' + log2)
+            jitstat[column_name] = jitstat.iloc[:, 1] / jitstat.iloc[:, 0]
+            # jitstat_count = jitstat.sort_values('1-' + log1, ascending=False).head(n)
+            # print(jitstat_count)
+        elif Type == "time":
+            if d1['time'].dtypes != np.dtype('float64'):
+                d1['time'] = d1['time'].astype(float)
+            if d2['time'].dtypes != np.dtype('float64'):
+                d2['time'] = d2['time'].astype(float)
+            jitstat = pd.concat((d1.groupby(name)['time'].sum(), d2.groupby(name)['time'].sum()), axis=1, sort=True)
+            jitstat.columns = ('1-' + log1, '2-' + log2)
+            jitstat[column_name] = jitstat.iloc[:, 1] / jitstat.iloc[:, 0]
+            jitstat_time = jitstat.sort_values('1-' + log1, ascending=False).head(n)
+            title = name + column_name + " Time Comparison"
+            jitstat_compare = jitstat_time.drop(columns=['1-' + log1, '2-' + log2])
+            jitstat_compare[:topk].plot.bar(
+                ax=ax, title=title, figsize=figsize, logx=False, rot=45)
+            filename = name + " Time Comparison"
+            ax.figure.savefig(filename)
+        return
+
+    def parse_raw_output_to_csv(self, filepath, csvpath='mkldnn_log.csv', keyword='dnnl_verbose'):
+
+        with open(csvpath, "w") as file:
+            with open(filepath) as fp:
+                line = fp.readline()
+                cnt = 1
+                while line:
+                    if line.find(keyword) != -1:
+                        file.write(line)
+                    line = fp.readline()
+                    cnt += 1
+        return csvpath
diff --git a/quickstart/common/pytorch/gpu/setvars.sh b/quickstart/common/pytorch/gpu/setvars.sh
old mode 100644
new mode 100755
diff --git a/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/DEVCATALOG.md b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/DEVCATALOG.md
index 66f816829..3efe492dc 100644
--- a/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/DEVCATALOG.md
+++ b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/DEVCATALOG.md
@@ -33,7 +33,6 @@ export IMAGE_NAME=intel/generative-ai:pytorch-flex-gpu-stable-diffusion-inferenc
 export PRECISION=<provide either fp32 or fp16>
 export OUTPUT_DIR=<path to output directory>
 export SCRIPT=quickstart/online_inference.sh
-export OUTPUT_DIR=<path to output directory>
 
 DOCKER_ARGS="--rm -it"
 
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
index 064a631e6..1d0b34b05 100644
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
@@ -69,6 +69,12 @@ IOMP should be installed in your conda env from the [General setup](#general-set
 > `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
 > otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
 
+## Calibration
+If user needs to re-do the calibration, here is the reference command:
+```
+DATASET_DIR=${DATASET_DIR} PRECISION=int8 CONFIG_FILE=${INT8_CONFIG_FILE} bash calibration.sh
+```
+
 ## Datasets
 
 ### ImageNet
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/calibration.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/calibration.sh
new file mode 100644
index 000000000..ac101a53f
--- /dev/null
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/calibration.sh
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/image_recognition/pytorch/common/main.py"  ]; then
+    echo "Could not find the script of main.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the main.py exist at the: \${MODEL_DIR}/models/image_recognition/pytorch/common/main.py"
+    exit 1
+fi
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+  exit 1
+fi
+
+if [ -z "${CONFIG_FILE}" ]; then
+  echo "The required environment variable CONFIG_FILE has not been set"
+  echo "Please set CONFIG_FILE to where to store the int8 config json."
+  exit 1
+fi
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  exit 1
+fi
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+BATCH_SIZE=128
+
+rm -rf ${OUTPUT_DIR}/resnet50_accuracy_log*
+
+if [[ "$PRECISION" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+# download pretrained weight.
+python ${MODEL_DIR}/models/image_recognition/pytorch/common/hub_help.py \
+    --url https://download.pytorch.org/models/resnet50-0676ba61.pth
+
+ARGS=""
+ARGS="$ARGS -e -a resnet50 ${DATASET_DIR}"
+
+echo "running int8 path"
+ARGS="$ARGS --int8 --configure-dir ${CONFIG_FILE}"
+
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator jemalloc \
+    ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+    $ARGS \
+    --ipex \
+    --pretrained \
+    -j 0 \
+    --calibration \
+    -b $BATCH_SIZE 2>&1 | tee ${OUTPUT_DIR}/resnet50_accuracy_log_${PRECISION}.log \
+
+wait
+
+accuracy=$(grep 'Accuracy:' ${OUTPUT_DIR}/resnet50_accuracy_log_${PRECISION}.log |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
+echo "resnet50;"accuracy";${PRECISION};${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
index c926d3675..451bb0ee5 100755
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
@@ -48,9 +48,16 @@ ARGS="$ARGS -e -a resnet50 ../ --dummy"
 # default value, you can fine-tune it to get perfect performance.
 BATCH_SIZE=112
 
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+
+
 if [[ $PRECISION == "int8" || $PRECISION == "avx-int8" ]]; then
-    BATCH_SIZE=116
     echo "running int8 path"
+    NUMA_NODES=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+    CORES_PER_NODE=`expr $TOTAL_CORES / $NUMA_NODES`
+    BATCH_SIZE=`expr $CORES_PER_NODE \* 8`
     ARGS="$ARGS --int8 --configure-dir ${MODEL_DIR}/models/image_recognition/pytorch/common/resnet50_configure_sym.json"
 elif [[ $PRECISION == "bf16" ]]; then
     BATCH_SIZE=68
@@ -74,12 +81,6 @@ else
     exit 1
 fi
 
-CORES=`lscpu | grep Core | awk '{print $4}'`
-SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-TOTAL_CORES=`expr $CORES \* $SOCKETS`
-
-CORES_PER_INSTANCE=$CORES
-
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
index 93ed35d09..e43421281 100755
--- a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
@@ -102,33 +102,56 @@ export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
 
-export CCL_MNIC=global
-export CCL_MNIC_NAME=rocep56s0,rocep73s0,rocep152s0,rocep216s0 #rocep56s0,rocep59s0,rocep73s0,rocep76s0,rocep152s0,rocep155s0,rocep216s0,rocep219s0
-export CCL_MNIC_COUNT=4
-export PSM3_PRINT_STATS=0
+#oneCCL settings
+export CCL_WORKER_COUNT=8
+export CCL_LOG_LEVEL=info
+export CCL_BF16=avx512bf
+export CCL_ATL_TRANSPORT=ofi
+export CCL_MNIC_COUNT=2
+export CCL_MNIC=local
+export CCL_MNIC_NAME=irdma1,irdma5
+export CCL_ALLREDUCE=ring
+export CCL_WORKER_COUNT=8
+
+for (( i = $SOCKETS; i < 2*$SOCKETS; i++ )); do  # pin CCL workers to HT
+  START_CORE=$(( i * CORES ))
+  for (( j = 0; j < $CCL_WORKER_COUNT; j++)); do
+   CCL_WORKER_AFFINITY="${CCL_WORKER_AFFINITY} $((START_CORE + j))"
+  done
+done
+
+export CCL_WORKER_AFFINITY=`echo ${CCL_WORKER_AFFINITY} | tr " " ","`
+
+
+#DDP settings
+export TORCH_CPP_LOG_LEVEL=INFO
+export TORCH_DISTRIBUTED_DEBUG=INFO
+export MASTER_ADDR=`head -1 hostfile`
+
+# Fabric settings
 export FI_PROVIDER=psm3
-export CCL_ALLREDUCE=rabenseifner
-export PSM3_IDENTIFY=1
 export PSM3_IDENTIFY=1
 export PSM3_ALLOW_ROUTERS=1
 export PSM3_RDMA=1
+export PSM3_PRINT_STATS=0
 export PSM3_RV_MR_CACHE_SIZE=8192
-export FI_PROVIDER_PATH=/usr/lib64/libfabric
+export PSM3_KASSIST_MODE=none
+#export PSM3_NIC='irdma*
+export FI_PSM3_CONN_TIMEOUT=100
 
-
-rm -rf ${OUTPUT_DIR}/resnet50_dist_training_log_*
+rm -rf ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}*
 
 oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
 source $oneccl_bindings_for_pytorch_path/env/setvars.sh
 
+export FI_PROVIDER_PATH=$oneccl_bindings_for_pytorch_path/lib/prov
+
 python -m intel_extension_for_pytorch.cpu.launch \
-    --memory-allocator jemalloc \
+    --memory-allocator tcmalloc \
     --distributed \
     --nnodes ${NNODES} \
     --hostfile ${HOSTFILE} \
-    --nproc_per_node ${SOCKETS} \
-    --ncore_per_instance ${CORES_PER_INSTANCE} \
-    --logical_core_for_ccl --ccl_worker_count 8 \
+    --logical_cores_for_ccl --ccl_worker_count 8 \
     ${MODEL_DIR}/models/image_recognition/pytorch/common/train.py \
     $ARGS \
     --epochs $TRAINING_EPOCHS \
@@ -144,7 +167,7 @@ python -m intel_extension_for_pytorch.cpu.launch \
 # For the summary of results
 wait
 
-throughput=$(grep 'Training throughput:' ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}.log |sed -e 's/.*Training throughput//;s/[^0-9.]//g' |awk '
+throughput=$(grep 'Training throughput(compute):' ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}.log |sed -e 's/.*Training throughput//;s/[^0-9.]//g' |awk '
 BEGIN {
        sum = 0;
 i = 0;
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_single_socket.sh b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_single_socket.sh
index 45cfb298a..bb7fdcd1c 100644
--- a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_single_socket.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_single_socket.sh
@@ -98,10 +98,10 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 rm -rf ./resnet50_training_log_*
 
 python -m intel_extension_for_pytorch.cpu.launch \
-    --memory-allocator jemalloc \
+    --memory-allocator tcmalloc \
     --ninstances 1 \
-    --ncore_per_instance ${CORES_PER_INSTANCE} \
-    --log_path=${OUTPUT_DIR} \
+    --ncores_per_instance ${CORES_PER_INSTANCE} \
+    --log_dir=${OUTPUT_DIR} \
     --log_file_prefix="./resnet50_training_log_${PRECISION}" \
     ${MODEL_DIR}/models/image_recognition/pytorch/common/train.py \
     $ARGS \
@@ -115,13 +115,14 @@ python -m intel_extension_for_pytorch.cpu.launch \
     --base-op=LARS \
     --base-lr 10.5 \
     --weight-decay 0.00005 \
+    --warmup-iterations 20 \
     -i 200 \
     --train-no-eval
 
 # For the summary of results
 wait
 
-throughput=$(grep 'Training throughput:'  ${OUTPUT_DIR}/resnet50_training_log_${PRECISION}_* |sed -e 's/.*throughput//;s/[^0-9.]//g' |awk '
+throughput=$(grep 'Training throughput(compute):'  ${OUTPUT_DIR}/resnet50_training_log_${PRECISION}_* |sed -e 's/.*throughput//;s/[^0-9.]//g' |awk '
 BEGIN {
     sum = 0;
     i = 0;
diff --git a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
index 2de069aba..33da70d0e 100644
--- a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
+++ b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
@@ -35,8 +35,8 @@ The folder that contains the `val` directory should be set as the
 | `inference_block_format.sh` | Runs ResNet50 V1.5 INT8 inference (block format)|
 
 Requirements:
-* Host machine has Intel(R) Data Center Max Series GPU
-* Follow instructions to install GPU-compatible driver [602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
+* Host machine has Intel(R) Data Center Max Series 1550 GPU x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
 * Docker
 
 ### Docker pull command:
@@ -51,12 +51,11 @@ export PRECISION=<export precision>
 export DATASET_DIR=${PWD}/imagenet
 export OUTPUT_DIR=${PWD}/logs
 export BATCH_SIZE=<export batch size,default is 1024>
-export NUM_ITERATIONS=<export number of iterations,default is 10>
+export NUM_ITERATIONS=<export number of iterations,default is 500>
 IMAGE_NAME=intel/image-recognition:pytorch-max-gpu-resnet50v1-5-inference
 DOCKER_ARGS="--rm -it"
-
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
 SCRIPT=quickstart/inference_block_format.sh
-Tile=2
 
 docker run \
   --privileged \
@@ -67,7 +66,7 @@ docker run \
   --env DATASET_DIR=${DATASET_DIR} \
   --env PRECISION=${PRECISION} \
   --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env Tile=${Tile} \
+  --env NUM_OAM=${NUM_OAM} \
   --env http_proxy=${http_proxy} \
   --env https_proxy=${https_proxy} \
   --env no_proxy=${no_proxy} \
@@ -77,4 +76,3 @@ docker run \
   $IMAGE_NAME \
   /bin/bash $SCRIPT
 ```
-
diff --git a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md
index 870207d95..aa9ee448c 100644
--- a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md
+++ b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md
@@ -5,12 +5,18 @@
 ## Description
 
 This document has instructions for running ResNet50v1.5 inference using
-Intel(R) Extension for PyTorch with GPU.
+Intel® Extension for PyTorch with GPU.
 
 <!--- 20. GPU Setup -->
 ## Software Requirements:
-- Intel® Data Center GPU Max Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/installation.html) to install the latest IPEX version and other prerequisites.
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-pytorch/) to install the latest IPEX version and other prerequisites.
 
 - Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
   - Intel® oneAPI DPC++ Compiler
@@ -65,19 +71,12 @@ The folder that contains the `val` directory should be set as the
 
 <!--- 50. Baremetal -->
 ## Run the model
-Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Clone the Model Zoo repository:
+* Clone the AI Reference Models repository:
   ```bash
   git clone https://github.com/IntelAI/models.git
   ```
-* Navigate to ResNet50v1.5 inference directory and install model specific dependencies for the workload:
+* Navigate to models directory:
   ```bash
-  # Navigate to the model zoo repo
   cd models
   ```
 
@@ -92,10 +91,12 @@ Set environment variables for the path to your dataset, an output directory to r
 # To run with ImageNet data, the dataset directory will need to be specified in addition to an output directory and precision.
 export DATASET_DIR=<path to the preprocessed imagenet dataset>
 export OUTPUT_DIR=<Path to save the output logs>
-export Tile=2
+export PRECISION=int8
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
 
 # Optional envs
-export BATCH_SIZE=<Set batch_size else it will run with default batch>
+export BATCH_SIZE=<Set batch_size else it will run with default batch of 1024>
+export NUM_ITERATIONS=<export number of iterations,default is 500>
 
 # Run a quickstart script
 ./quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh
diff --git a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh
index 7e3747467..816353358 100755
--- a/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh
+++ b/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh
@@ -19,12 +19,6 @@ MODEL_DIR=${MODEL_DIR-$PWD}
 NUM_ITERATIONS=${NUM_ITERATIONS-500}
 BATCH_SIZE=${BATCH_SIZE-1024}
 
-if [[ -z "${Tile}" ]]; then
-    Tile=${Tile-1}
-else
-    Tile=${Tile}
-fi
-
 dataset_arg="${DATASET_DIR}"
 if [[ -z "${DATASET_DIR}" ]]; then
   echo "Using Dummy data since environment variable DATASET_DIR has not been set"
@@ -41,6 +35,7 @@ fi
 
 # Create the output directory, if it doesn't already exist
 mkdir -p $OUTPUT_DIR
+declare -a str
 
 export OverrideDefaultFP64Settings=1 
 export IGC_EnableDPEmulation=1
@@ -49,33 +44,7 @@ export IPEX_ONEDNN_LAYOUT=1
 export IPEX_LAYOUT_OPT=1
 export IPEX_XPU_ONEDNN_LAYOUT=1
 
-
-resnet50_log_analysis() {
-    # $1 : src raw log
-    # $2 : dst format log
-    # $3 : inference or training
-    # $4 : bs
-
-    bs=$4
-    if [ -f $2 ]; then
-        rm $2
-    fi
-
-    if [ "inference" == "$3" ]; then
-        echo -e 'Batch Size: ' $bs >$2
-        cat $1 | grep Test | tail -n6 | head -n5 |
-            awk -v bs=${bs} -F ' ' '{a+=$5}END{printf "Performance Benchmark Time: %.3f sec, Throughput: %.2f FPS\n", a/5, bs*5/a}' >>$2
-        cat $1 | tail -n2 | grep "Acc@1" | awk -F ' ' '{printf "Accuracy: acc@1 %.2f\n", $3}' >>$2
-    elif [ "training" == "$3" ]; then
-        echo -e 'Batch Size: ' $bs >$2
-        cat $1 | grep Epoch | tail -n1 | awk -v bs=${bs} -F ' ' '{printf "Performance Benchmark Time: %.3f sec, Throughput: %.2f FPS\n", $5, bs/$4}' >>$2
-    else
-        echo -e 'Invalid input! Only inference or training are supported.'
-        exit 0
-    fi
-}
-
-if [[ ${Tile} == "1" ]]; then
+if [[ -z "${NUM_OAM}" ]]; then
     # int8 uses a different python script
     echo "resnet50 int8 inference block"
     IPEX_XPU_ONEDNN_LAYOUT=1 python -u models/image_recognition/pytorch/resnet50v1_5/inference/gpu/main.py \
@@ -88,33 +57,30 @@ if [[ ${Tile} == "1" ]]; then
         --num-iterations ${NUM_ITERATIONS} \
         --benchmark 1 \
         ${dataset_arg}  2>&1 | tee ${OUTPUT_DIR}//resnet50_int8_inf_block_t0_raw.log
-    resnet50_log_analysis ${OUTPUT_DIR}/resnet50_int8_inf_block_t0_raw.log ${OUTPUT_DIR}/resnet50_int8_inf_block_t0.log inference ${BATCH_SIZE}
-elif [[ ${Tile} == "2" ]]; then
-    echo "resnet50 int8 inference block two tile"
-    ZE_AFFINITY_MASK=0.0 IPEX_XPU_ONEDNN_LAYOUT=1 python -u models/image_recognition/pytorch/resnet50v1_5/inference/gpu/main.py \
-        -a resnet50 \
-        -b ${BATCH_SIZE} \
-        --xpu 0 \
-        -e \
-        --pretrained \
-        --int8 1 \
-        --num-iterations ${NUM_ITERATIONS} \
-        --benchmark 1 \
-        ${dataset_arg}  2>&1 | tee ${OUTPUT_DIR}//resnet50_int8_inf_block_t0_raw.log &
-    ZE_AFFINITY_MASK=0.1 IPEX_XPU_ONEDNN_LAYOUT=1 python -u models/image_recognition/pytorch/resnet50v1_5/inference/gpu/main.py \
-        -a resnet50 \
-        -b ${BATCH_SIZE} \
-        --xpu 0 \
-        -e \
-        --pretrained \
-        --int8 1 \
-        --num-iterations ${NUM_ITERATIONS} \
-        --benchmark 1 \
-        ${dataset_arg}  2>&1 | tee ${OUTPUT_DIR}//resnet50_int8_inf_block_t1_raw.log
-    resnet50_log_analysis ${OUTPUT_DIR}/resnet50_int8_inf_block_t0_raw.log ${OUTPUT_DIR}/resnet50_int8_inf_block_t0.log inference ${BATCH_SIZE}
-    resnet50_log_analysis ${OUTPUT_DIR}/resnet50_int8_inf_block_t1_raw.log ${OUTPUT_DIR}/resnet50_int8_inf_block_t1.log inference ${BATCH_SIZE}
+
+elif [[ ${NUM_OAM} == "4" ]]; then
+    echo "resnet50 int8 inference block on ${NUM_OAM} OAM module(s)"
+    NUM_TILES_PER_GPU=2
+    for i in $( eval echo {0..$((NUM_OAM-1))} )
+        do
+            for j in $( eval echo {0..$((NUM_TILES_PER_GPU-1))} )
+                do
+                    str+=("ZE_AFFINITY_MASK="${i}"."${j}" IPEX_XPU_ONEDNN_LAYOUT=1 python -u models/image_recognition/pytorch/resnet50v1_5/inference/gpu/main.py \
+                            -a resnet50 \
+                            -b ${BATCH_SIZE} \
+                            --xpu 0 \
+                            -e \
+                            --pretrained \
+                            --int8 1 \
+                            --num-iterations ${NUM_ITERATIONS} \
+                            --benchmark 1 \
+                            ${dataset_arg}  2>&1 | tee ${OUTPUT_DIR}/resnet50_int8_inf_block_c${i}_t${j}_raw.log & ")
+                done
+        done
+    str=${str[@]}
+    cmd_line=${str::-2}
+    eval $cmd_line
 else
-    echo "The specified Tile '${Tile}' is unsupported."
-    echo "Supported tile number are: 1 and 2"
+    echo "Currently only x4 OAM Modules are supported"
     exit 1
 fi
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_SPR_baremetal.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_SPR_baremetal.md
index 560e0c276..c792d16ac 100644
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_SPR_baremetal.md
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_SPR_baremetal.md
@@ -87,9 +87,11 @@ export PRETRAINED_MODEL=<path to the downloaded pretrained model file>
 export OUTPUT_DIR=<directory where log files will be written>
 
 # Optional env vars
-export BATCH_SIZE=<customized batch size value>
-export WARMUP_STEPS=<customized warm up steps value>
-export STEPS=<customized steps value>
+export BATCH_SIZE=<customized batch size value, otherwise it will run with the default value>
+export WARMUP_STEPS=<customized warm up steps value, otherwise it will run with the default value>
+export STEPS=<customized steps value, otherwise it will run with the default value>
+export OMP_NUM_THREADS=<customized value for omp_num_threads, otherwise it will run with the default value>
+export CORES_PER_INSTANCE=<customized value for cores_per_instance, otherwise it will run with the default value>
 ```
 ### Run real time inference (Latency):
 ```
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
index 04a2ac9ed..431e298c6 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
@@ -82,6 +82,9 @@ MODE="inference"
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 BATCH_SIZE="${BATCH_SIZE:-"100"}"
 
+# Remove old log file
+rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_accuracy.log
+
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
index ef301ee2a..b1f56af2b 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
@@ -68,8 +68,19 @@ elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
 fi
 
 MODE="inference"
-# Use 4core/instance 
-CORES_PER_INSTANCE="4"
+
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE="4"
+  echo "Running with default ${CORES_PER_INSTANCE} cores per instance"
+fi
+
+# If OMP_NUM_THREADS env is not mentioned, Get number of cores per instance
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=4
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 
 #Set up env variable for bfloat32
 if [[ $PRECISION == "bfloat32" ]]; then
@@ -98,6 +109,9 @@ echo "WARMUP_STEPS: $WARMUP_STEPS"
 export TF_ENABLE_MKL_NATIVE_FORMAT=1 
 export TF_ONEDNN_ENABLE_FAST_CONV=1 
 
+# Remove old log file
+rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log
+
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh
index 6658cd560..6fa6194d5 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh
@@ -67,8 +67,14 @@ elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
   exit 1
 fi
 
-# Get number of cores per socket line from lscpu
-export OMP_NUM_THREADS=4
+cores_per_socket="4"
+
+# If OMP_NUM_THREADS env is not mentioned, then run with the default value
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=4
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 
 MODE="inference"
 
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
index 40c9a23b1..02a821aad 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
@@ -71,7 +71,22 @@ export TF_ENABLE_MKL_NATIVE_FORMAT=1
 export TF_ONEDNN_ENABLE_FAST_CONV=1
 
 MODE="inference"
-CORES_PER_INSTANCE="socket"
+  
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE="socket"
+  echo "Runs an instance per ${CORES_PER_INSTANCE}"
+fi
+
+cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
+cores_per_socket="${cores_per_socket//[[:blank:]]/}"
+
+# If OMP_NUM_THREADS env is not mentioned, then run with the default value
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=4
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 
 #Set up env variable for bfloat32
 if [[ $PRECISION == "bfloat32" ]]; then
@@ -79,10 +94,6 @@ if [[ $PRECISION == "bfloat32" ]]; then
   PRECISION="fp32"
 fi
 
-# Get number of cores per socket line from lscpu
-cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
-cores_per_socket="${cores_per_socket//[[:blank:]]/}"
-
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [[ $PRECISION == "bfloat16" ]]; then
   BATCH_SIZE="${BATCH_SIZE:-"80"}"
@@ -106,6 +117,9 @@ else
 fi
 echo "WARMUP_STEPS: $WARMUP_STEPS"
 
+# Remove old log file
+rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log
+
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
index fcfb9d7ae..09bb2821e 100644
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md
@@ -15,13 +15,13 @@ Set the `DATASET_DIR` to point to the TF records directory when running ResNet50
 
 | Script name | Description |
 |-------------|-------------|
-| `batch_inference` | Runs ResNet50 V1.5 batch inference for the precision set |
+| `batch_inference` | Runs ResNet50 V1.5 batch inference for the precision set on x4 OAM |
 | `accuracy` | Measures model accuracy for the precision set |
 | `online_inference` | Runs ResNet50 V1.5 online inference for the precision set |
 
 Requirements:
-* Host machine has Intel® Data Center Max Series GPU
-* Follow instructions to install GPU-compatible driver [602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
+* Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
 * Docker
 
 ### Docker pull command:
@@ -29,23 +29,18 @@ Requirements:
 ```
 docker pull intel/image-recognition:tf-max-gpu-resnet50v1-5-inference
 ```
-The ResNet50 v1.5 inference container includes scripts,model and libraries need to run INT8,FP32 and FP16 inference. To run the quickstart scripts using this container, you'll need to provide volume mounts for the ImageNet dataset. For running accuracy test, you will need to provide the `DATASET_DIR` to point to the pre-processed ImageNet Dataset. For batch and online inference, the script uses dummy data. You will need to provide an output directory where log files will be written. 
+The ResNet50 v1.5 inference container includes scripts,model and libraries need to run int8,fp32,fp16 inference. To run the quickstart scripts using this container, you'll need to provide volume mounts for the ImageNet dataset. For running accuracy test, you will need to provide the `DATASET_DIR` to point to the pre-processed ImageNet Dataset. For batch and online inference, the script uses dummy data. You will need to provide an output directory where log files will be written. 
 
 ```
-export PRECISION=int8
+export PRECISION=<provide int8,fp32 or fp16 precision>
 export OUTPUT_DIR=<path to output directory>
 export DATASET_DIR=<path to the preprocessed imagenet dataset>
 IMAGE_NAME=intel/image-recognition:tf-max-gpu-resnet50v1-5-inference
 DOCKER_ARGS="--rm -it"
 export SCRIPT=batch_inference.sh 
-export FROZEN_GRAPH=/workspace/tf-pvc-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-${PRECISION}-gpu.pb
+export FROZEN_GRAPH=/workspace/tf-max-series-resnet50v1-5-inference/pretrained_models/resnet50v1_5-frozen_graph-${PRECISION}-gpu.pb
 export GPU_TYPE=max_series
-
-if [[ ${SCRIPT} == batch_inference.sh ]]; then
-   export Tile=2
-else
-   export Tile=1
-fi
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
 
 docker run \
   --privileged \
@@ -54,10 +49,10 @@ docker run \
   --privileged \
   --env PRECISION=${PRECISION} \
   --env GPU_TYPE=${GPU_TYPE} \
+  --env NUM_OAM=${NUM_OAM} \
   --env OUTPUT_DIR=${OUTPUT_DIR} \
   --env DATASET_DIR=${DATASET_DIR} \
   --env FROZEN_GRAPH=${FROZEN_GRAPH} \
-  --env Tile=${Tile} \
   --env http_proxy=${http_proxy} \
   --env https_proxy=${https_proxy} \
   --env no_proxy=${no_proxy} \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Max_Series.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Max_Series.md
index 32fa8f6e5..4a3f25655 100644
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Max_Series.md
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Max_Series.md
@@ -9,8 +9,14 @@ Intel® Extension for TensorFlow with Intel® Data Center GPU Max Series.
 
 <!--- 20. GPU Setup -->
 ## Software Requirements:
-- Intel® Data Center GPU Max Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-tensorflow/latest/get_started.html) to install the latest ITEX version and other prerequisites.
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-tensorflow/) to install the latest ITEX version and other prerequisites.
 
 - Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
   - Intel® oneAPI DPC++ Compiler
@@ -41,20 +47,14 @@ Set the `DATASET_DIR` to point to the TF records directory when running ResNet50
 ## Quick Start Scripts
 
 | Script name | Description |
-|:-------------:|:-------------:|
-| [`online_inference.sh`](online_inference.sh) | Runs online inference for int8, fp16 and fp32 precisions. | 
-| [`batch_inference.sh`](batch_inference.sh)| Runs batch inference for int8, fp16 and fp32 precisions. |
-| [`accuracy.sh`](accuracy.sh) | Measures the model accuracy for int8, fp16 and fp32 precisions. |
+|-------------|-------------|
+| `batch_inference` | Runs ResNet50 V1.5 batch inference for the precision set on x4 OAM |
+| `accuracy` | Measures model accuracy for the precision set |
+| `online_inference` | Runs ResNet50 V1.5 online inference for the precision set |
 
 <!--- 50. Baremetal -->
 ## Run the model
 Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-  
 * Download the frozen graph model file, and set the FROZEN_GRAPH environment variable to point to where it was saved:
   ```bash
   # For fp32 and fp16:
@@ -74,23 +74,25 @@ TF records files will need to be set as the `DATASET_DIR` environment variable
 prior to running a [quickstart script](#quick-start-scripts).
 
 ### Run the model on Baremetal
-Navigate to the ResNet50 v1.5 inference directory, and set environment variables:
+Navigate to the models directory, and set environment variables:
 ```
 cd models
 export OUTPUT_DIR=<path where output log files will be written>
 export PRECISION=<Set precision: int8, fp16 or fp32>
 export FROZEN_GRAPH=<path to pretrained model file (*.pb)>
 export GPU_TYPE=max_series
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
 
 # Optional envs
 export BATCH_SIZE=<Set batch_size else it will run with default batch>
 
-# Set 'Tile' env variable is only for running "batch_inference.sh" script:
-export Tile=2
-
 # Set 'DATASET_DIR' only when running "accuracy.sh" script, "online_inference.sh" and "batch_inference.sh" scripts run with synthetic dataset:
 export DATASET_DIR=<path to the preprocessed imagenet dataset directory>
 
+# Install dependencies:
+NOTE: To run apt-get commands may require sudo root permissions.
+./quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/setup.sh
+
 # Run quickstart script:
 ./quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/<script name>.sh
 ```
@@ -99,4 +101,3 @@ export DATASET_DIR=<path to the preprocessed imagenet dataset directory>
 ## License
 
 [LICENSE](/LICENSE)
-
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/accuracy.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/accuracy.sh
index 2857cadaa..c969ce1e6 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/accuracy.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/accuracy.sh
@@ -95,20 +95,23 @@ elif [[ $GPU_TYPE == "max_series" ]]; then
 fi
 
 if [[ $PRECISION == "fp16" ]]; then
+  DTYPE="float16"
   export ITEX_AUTO_MIXED_PRECISION=1
   export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
 fi
 
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=resnet50v1_5 \
-         --precision=${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${pretrained_model} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
+if [[ $PRECISION == "int8" ]]; then
+  DTYPE="int8"
+fi
+
+if [[ $PRECISION == "fp32" ]]; then
+  DTYPE="float32"
+fi
+
+python -u models/image_recognition/tensorflow/resnet50v1_5/inference/gpu/int8/eval_image_classifier_inference.py \
+         --input-graph=${pretrained_model} \
+         --data-num-inter-threads 1 \
          --accuracy-only \
-         --batch-size=${BATCH_SIZE} \
-         --gpu \
-         $@
+	 --batch-size=${BATCH_SIZE} \
+         --data-location ${DATASET_DIR} \
+         --dtype ${DTYPE} 2>&1 | tee ${OUTPUT_DIR}/resnet50_acc_${PRECISION}_raw.log
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/batch_inference.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/batch_inference.sh
index fdafe9bdb..0a229de5a 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/batch_inference.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/batch_inference.sh
@@ -42,6 +42,7 @@ done
 
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}
+declare -a str
 
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}"]; then
@@ -62,7 +63,6 @@ if [[ $GPU_TYPE == "flex_series" ]]; then
     else
       pretrained_model=${FROZEN_GRAPH}
     fi
-    # WARMUP="-- warmup_steps=10 steps=500"
   else 
     echo "FLEX SERIES GPU SUPPORTS ONLY INT8 PRECISION"
     exit 1
@@ -75,7 +75,6 @@ elif [[ $GPU_TYPE == "max_series" ]]; then
     else
       pretrained_model=${FROZEN_GRAPH}
     fi
-    WARMUP="-- warmup_steps=5 steps=20 disable-tcmalloc=True"
   else 
     echo "MAX SERIES GPU SUPPORTS ONLY INT8, FP32 AND FP16 PRECISION"
     exit 1
@@ -83,18 +82,23 @@ elif [[ $GPU_TYPE == "max_series" ]]; then
 fi
 
 if [[ $PRECISION == "fp16" ]]; then
+  DTYPE="float16"
   export ITEX_AUTO_MIXED_PRECISION=1
   export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
 fi
 
-if [[ -z "${Tile}" ]]; then
-    Tile=${Tile-1}
+if [[ $PRECISION == "int8" ]]; then
+  DTYPE="int8"
+  benchmark="--benchmark"
 else
-    Tile=${Tile}
+  benchmark=""
 fi
 
-# source "${MODEL_DIR}/quickstart/common/utils.sh"
-if [[ ${Tile} == "1" ]]; then
+if [[ $PRECISION == "fp32" ]]; then
+  DTYPE="float32"
+fi
+
+if [[ -z "${NUM_OAM}" ]]; then
     echo "resnet50 v1.5 int8 inference"
          mac=`lspci | grep Dis| head -n 1| awk '{print $1}'`
          node=`lspci -s $mac -v | grep NUMA | awk -F, '{print $5}' | awk '{print $3}'`
@@ -102,48 +106,28 @@ if [[ ${Tile} == "1" ]]; then
          --input-graph=${pretrained_model} \
          --warmup-steps=10 \
          --steps=500 \
-         --batch-size=1024 \
-         --benchmark 
-        #  numactl -N $node -l python benchmarks/launch_benchmark.py \
-        #  --model-name=resnet50v1_5 \
-        #  --precision=${PRECISION} \
-        #  --mode=inference \
-        #  --framework tensorflow \
-        #  --in-graph ${pretrained_model} \
-        #  --output-dir ${OUTPUT_DIR} \
-        #  --batch-size=${BATCH_SIZE} \
-        #  --benchmark-only \
-        #  --gpu \
-        #  $@ \
-        #  ${WARMUP} 2>&1 | tee ${OUTPUT_DIR}//resnet50_${PRECISION}_inf_t0_raw.log
-
-elif [[ ${Tile} == "2" ]]; then
-        echo "resnet50 v1.5 int8 two-tile inference"
-        ZE_AFFINITY_MASK=0.0 python benchmarks/launch_benchmark.py \
-         --model-name=resnet50v1_5 \
-         --precision=${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${pretrained_model} \
-         --output-dir ${OUTPUT_DIR} \
-         --batch-size=${BATCH_SIZE} \
-         --benchmark-only \
-         --gpu \
-         $@ \
-         ${WARMUP} 2>&1 | tee ${OUTPUT_DIR}//resnet50_${PRECISION}_inf_t0_raw.log &
-         ZE_AFFINITY_MASK=0.1 python benchmarks/launch_benchmark.py \
-         --model-name=resnet50v1_5 \
-         --precision=${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${pretrained_model} \
-         --output-dir ${OUTPUT_DIR} \
          --batch-size=${BATCH_SIZE} \
-         --benchmark-only \
-         --gpu \
-         $@ \
-         ${WARMUP} 2>&1 | tee ${OUTPUT_DIR}//resnet50_${PRECISION}_inf_t1_raw.log 
+         --benchmark 
+elif [[ ${NUM_OAM} == "4" ]]; then
+        echo "resnet50 v1.5 int8 inference on ${NUM_OAM} OAM modules"
+        NUM_TILES_PER_GPU=2
+        for i in $( eval echo {0..$((NUM_OAM-1))} )
+          do
+            for j in $( eval echo {0..$((NUM_TILES_PER_GPU-1))} )
+                do
+                  str+=("ZE_AFFINITY_MASK="${i}"."${j}" python -u models/image_recognition/tensorflow/resnet50v1_5/inference/gpu/int8/eval_image_classifier_inference.py \
+                        --input-graph=${pretrained_model} \
+                        --warmup-steps=5 \
+                        --steps=20 \
+                        --batch-size=${BATCH_SIZE} \
+                        --dtype ${DTYPE} \
+                        ${benchmark} 2>&1 | tee ${OUTPUT_DIR}/resnet50_inf_${PRECISION}_c${i}_t${j}_raw.log & ")
+                done
+          done
+        str=${str[@]}
+        cmd_line=${str::-2}
+        eval $cmd_line
 else
-    echo"Only Tiles 1 and 2 supported."
+    echo "Currently only x4 OAM Modules are supported"
     exit 1
 fi
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/online_inference.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/online_inference.sh
index 5c8581502..d27c5dbc9 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/online_inference.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/online_inference.sh
@@ -84,16 +84,28 @@ elif [[ $GPU_TYPE == "max_series" ]]; then
 fi
 
 if [[ $PRECISION == "fp16" ]]; then
+  DTYPE="float16"
   export ITEX_AUTO_MIXED_PRECISION=1
   export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
 fi
 
+if [[ $PRECISION == "int8" ]]; then
+  DTYPE="int8"
+  benchmark="--benchmark"
+else
+  benchmark=""
+fi
+
+if [[ $PRECISION == "fp32" ]]; then
+  DTYPE="float32"
+fi
 # source "${MODEL_DIR}/quickstart/common/utils.sh"
 mac=`lspci | grep Dis| head -n 1| awk '{print $1}'`
 node=`lspci -s $mac -v | grep NUMA | awk -F, '{print $5}' | awk '{print $3}'`
 numactl -N $node -l python -u models/image_recognition/tensorflow/resnet50v1_5/inference/gpu/int8/eval_image_classifier_inference.py \
-         --input-graph=${pretrained_model} \
-         --warmup-steps=10 \
-         --steps=5000 \
-         --batch-size=1 \
-         --benchmark 
+                        --input-graph=${pretrained_model} \
+                        --warmup-steps=5 \
+                        --steps=20 \
+                        --batch-size=$BATCH_SIZE \
+                        --dtype ${DTYPE} \
+                        ${benchmark} 2>&1 | tee ${OUTPUT_DIR}/resnet50_inf_bs${BATCH_SIZE}_${PRECISION}_raw.log
diff --git a/models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/__init__.py b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/setup.sh
old mode 100644
new mode 100755
similarity index 82%
rename from models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/__init__.py
rename to quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/setup.sh
index ce0a27f82..3cdcae5c1
--- a/models/image_recognition/tensorflow/vision_transformer/inference/bfloat16/__init__.py
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/setup.sh
@@ -1,5 +1,4 @@
-#
-# -*- coding: utf-8 -*-
+#!/usr/bin/env bash
 #
 # Copyright (c) 2023 Intel Corporation
 #
@@ -16,5 +15,5 @@
 # limitations under the License.
 #
 
-#
-
+apt-get update && \
+apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl 
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh
index 2939b5ca5..05733631e 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh
@@ -49,7 +49,12 @@ fi
 cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
 cores_per_socket="${cores_per_socket//[[:blank:]]/}"
 
-NUM_INSTANCES="1"
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+CORES_PER_NUMA=`expr $CORES \* $SOCKETS / $NUMAS`
+
+NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
 
 #Set up env variable for bfloat32
 if [[ $PRECISION == "bfloat32" ]]; then
@@ -74,9 +79,9 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --data-location=${DATASET_DIR} \
   --output-dir ${OUTPUT_DIR} \
   --mpi_num_processes=${NUM_INSTANCES} \
-  --mpi_num_processes_per_socket=1 \
+  --mpi_num_processes_per_socket=${NUM_INSTANCES} \
   --batch-size ${BATCH_SIZE} \
-  --num-intra-threads $cores_per_socket \
+  --num-intra-threads $CORES_PER_NUMA \
   --num-inter-threads 2 \
   $@ \
   -- \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh
index 472127c3e..ff4c2534a 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh
@@ -62,6 +62,9 @@ if [ -z "${BATCH_SIZE}"]; then
   echo "Running with default batch size of ${BATCH_SIZE}"
 fi
 
+# Remove old log file
+rm -rf  ${OUTPUT_DIR}/resnet50v1_5_fp32_training_bs${BATCH_SIZE}_all_instances.log
+
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _command python benchmarks/launch_benchmark.py \
   --model-name=resnet50v1_5 \
diff --git a/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/inference.sh b/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/inference.sh
index 316e98f3f..c0652804c 100755
--- a/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/inference.sh
+++ b/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/inference.sh
@@ -27,7 +27,6 @@ echo 'GPU_TYPE='$GPU_TYPE
 declare -A input_envs
 input_envs[PRECISION]=${PRECISION}
 input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
-input_envs[PRETRAINED_DIR]=${PRETRAINED_DIR}
 input_envs[BATCH_SIZE]=${BATCH_SIZE}
 input_envs[GPU_TYPE]=${GPU_TYPE}
 
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md
index 551d7e7c0..bef9a027c 100644
--- a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md
@@ -38,11 +38,11 @@ BERT_WEIGHT=$(pwd)/squad_large_finetuned_checkpoint
 
 | Script name | Description |
 |-------------|-------------|
-| `fp16_inference_plain_format.sh` | Runs BERT Large inference (plain format) for fp16 precision |
+| `fp16_inference_plain_format.sh` | Runs BERT Large inference (plain format) for FP16 precision |
 
 Requirements:
-* Host machine has Intel(R) Data Center Max Series GPU
-* Follow instructions to install GPU-compatible driver [602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
+* Host machine has Intel(R) Data Center Max Series 1550 x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
 * Docker
 
 ## Docker pull Command
@@ -56,13 +56,14 @@ The BERT Large inference container includes scripts,models,libraries needed to r
 ```
 export DATASET_DIR=<path to dataset>
 export OUTPUT_DIR=<path to output log files>
+export PRECISION=<export precision. FP16 is currently supported>
 export BERT_WEIGHT=$(pwd)/squad_large_finetuned_checkpoint
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export BATCH_SIZE=<provide batch size. default is 64>
 
 DOCKER_ARGS="--rm -it"
 IMAGE_NAME=intel/language-modeling:pytorch-max-gpu-bert-large-inference
-
 SCRIPT=quickstart/fp16_inference_plain_format.sh
-Tile=2
 
 docker run \
   --privileged \
@@ -71,7 +72,9 @@ docker run \
   --env DATASET_DIR=${DATASET_DIR} \
   --env BERT_WEIGHT=${BERT_WEIGHT} \
   --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env Tile=${Tile} \
+  --env NUM_OAM=${NUM_OAM} \
+  --env PRECISION=${PRECISION} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
   --env http_proxy=${http_proxy} \
   --env https_proxy=${https_proxy} \
   --env no_proxy=${no_proxy} \
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md
index 93b84c2b3..74970a7c2 100644
--- a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md
@@ -1,16 +1,22 @@
 <!--- 0. Title -->
-# BERT Large inference for Intel(R) Data Center GPU Max Series
+# BERT Large Inference
 
 <!-- 10. Description -->
 ## Description
 
 This document has instructions for running BERT Large inference using
-Intel-optimized PyTorch with Intel(R) Data Center GPU Max Series.
+Intel-optimized PyTorch with Intel® Data Center GPU Max Series.
 
 <!--- 20. GPU Setup -->
 ## Software Requirements:
-- Intel® Data Center GPU Max Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/installation.html) to install the latest IPEX version and other prerequisites.
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-pytorch/) to install the latest IPEX version and other prerequisites.
 
 - Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
   - Intel® oneAPI DPC++ Compiler
@@ -69,19 +75,12 @@ BERT_WEIGHT=$(pwd)/squad_large_finetuned_checkpoint
 
 <!--- 50. Baremetal -->
 ## Run the model
-Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Clone the Model Zoo repository:
+* Clone the Intel AI reference Models repository repository:
   ```bash
   git clone https://github.com/IntelAI/models.git
   ```
 * Navigate models directory and install model specific dependencies for the workload:
   ```bash
-  # Navigate to the model zoo repo
   cd models
   # Install model specific dependencies:
   python -m pip install -r models/language_modeling/pytorch/bert_large/inference/gpu/requirements.txt
@@ -95,7 +94,11 @@ downloading the SQuAD dataset.
 export DATASET_DIR=<path the dataset directory>
 export OUTPUT_DIR=<directory where log files will be written>
 export BERT_WEIGHT=<directory where BERT weight files will be downloaded>
-export Tile=2
+export PRECISION=<export precision. FP16 is currently supported>
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+
+# Optional envs
+export BATCH_SIZE=<Set batch_size else it will run with default batch of 64>
 
 # Run a quickstart script
 ./quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh
@@ -104,4 +107,4 @@ export Tile=2
 <!--- 80. License -->
 ## License
 
-[LICENSE](/LICENSE)
+[LICENSE](/LICENSE)
\ No newline at end of file
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/build.sh b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/build.sh
deleted file mode 100755
index 5242b699e..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/build.sh
+++ /dev/null
@@ -1,34 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2021-2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-PYTORCH_BASE_IMAGE=${PYTORCH_BASE_IMAGE:-intel/intel-extension-for-pytorch}
-PYTORCH_BASE_TAG=${PYTORCH_BASE_TAG:-xpu-max}
-IMAGE_NAME=${IMAGE_NAME:-intel/language-modeling:pytorch-max-gpu-bert-large-inference}
-
-docker build \
-    --build-arg PACKAGE_DIR=model_packages \
-    --build-arg PACKAGE_NAME=pytorch-max-series-bert-large-inference \
-    --build-arg MODEL_WORKSPACE=/workspace \
-    --build-arg http_proxy=$http_proxy \
-    --build-arg https_proxy=$https_proxy \
-    --build-arg no_proxy=$no_proxy \
-    --build-arg PYTORCH_BASE_IMAGE=${PYTORCH_BASE_IMAGE} \
-    --build-arg PYTORCH_BASE_TAG=${PYTORCH_BASE_TAG} \
-    -t $IMAGE_NAME \
-    -f pytorch-max-series-bert-large-inference.Dockerfile .
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh
index 676142bea..0dec2abd3 100755
--- a/quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh
@@ -18,14 +18,6 @@
 MODEL_DIR=${MODEL_DIR-$PWD}
 BATCH_SIZE=${BATCH_SIZE-64}
 
-#source ${MODEL_DIR}/quickstart/setvars.sh
-
-if [[ -z "${Tile}" ]]; then
-    Tile=${Tile-1}
-else
-    Tile=${Tile}
-fi
-
 if [[ -z "${DATASET_DIR}" ]]; then
   echo "The required environment variable DATASET_DIR has not been set"
   exit 1
@@ -51,70 +43,49 @@ if [[ -z $OUTPUT_DIR ]]; then
   exit 1
 fi
 
+if [[ -z $PRECISION ]]; then
+  echo "The required environment variable PRECISION has not been set"
+  exit 1
+fi
+
+if [[ -z $NUM_OAM ]]; then
+  echo "The required environment variable NUM_OAM has not been set."
+  exit 1
+fi
 # Create the output directory, if it doesn't already exist
 mkdir -p $OUTPUT_DIR
 
-bertsquad_log_analysis() {
-    # $1 : src raw log
-    # $2 : dst format log
-    # $3 : inference or training
-    # $4 : bs
-
-    if [ -f $2 ]; then
-        rm $2
-    fi
-
-    bs=$4
-
-    if [ "inference" == "$3" ]; then
-        echo -e 'Batch Size: ' $bs >$2
-        cat $1 | grep latency | tail -n6 | head -n4 |
-            awk -v bs=${bs} -F ' ' '{sum+=$8} END{printf "Performance Benchmark Time: %.3f sec, Throughput: %.2f seq/sec\n", sum/4, bs*4/sum}' >>$2
-        grep "\"f1\": " $1 | awk -F ' ' '{printf "Accuracy: f1 %.4f\n", $NF}' >>$2
-    elif [ "training" == "$3" ]; then
-        # only for fine tune (accuracy only)
-        echo -e 'Batch Size: ' $bs >$2
-        echo -e 'Performance Benchmark Time: N/A' >>$2
-        grep "\"f1\": " $1 | awk -F ' ' '{printf "Accuracy: f1 %.4f\n", $NF}' >>$2
-    else
-        echo -e 'Invalid input! Only inference or training are supported.'
-        exit 0
-    fi
-}
+if [[ ${PRECISION} != "FP16" ]]; then
+  echo "BERT Large Inference currently supports FP16 precision"
+  exit 1
+fi
 
-if [[ ${Tile} == "1" ]]; then
-  echo "bertsquad fp16 inference plain nchw"
+if [[ ${NUM_OAM} == "4" ]]; then
+  echo "bertsquad fp16 inference plain nchw on ${NUM_OAM} OAM modules"
+  NUM_TILES_PER_GPU=2
   cd ${MODEL_DIR}/models/language_modeling/pytorch/bert_large/inference/gpu/
-  bash cmd_infer.sh \
-      -m bert_large \
-      -d xpu \
-      -b $BATCH_SIZE \
-      -t FP16 \
-      -o None 2>&1 | tee ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0_raw.log
-  wait
-  bertsquad_log_analysis ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0_raw.log ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0.log inference ${BATCH_SIZE}
-  cd -
-elif [[ ${Tile} == "2" ]]; then
-  echo "bertsquad fp16 inference plain nchw 2 tile"
-  cd ${MODEL_DIR}/models/language_modeling/pytorch/bert_large/inference/gpu/
-  ZE_AFFINITY_MASK=0.0 bash cmd_infer.sh \
-      -m bert_large \
-      -d xpu \
-      -b $BATCH_SIZE \
-      -t FP16 \
-      -o None 2>&1 | tee ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0_raw.log &
-  ZE_AFFINITY_MASK=0.1 bash cmd_infer.sh \
-      -m bert_large \
-      -d xpu \
-      -b $BATCH_SIZE \
-      -t FP16 \
-      -o None 2>&1 | tee ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t1_raw.log
+  for i in $( eval echo {0..$((NUM_OAM-1))} )
+    do
+        for j in $( eval echo {0..$((NUM_TILES_PER_GPU-1))} )
+            do
+              str+=("ZE_AFFINITY_MASK="${i}"."${j}" bash cmd_infer.sh \
+                    -m bert_large \
+                    -d xpu \
+                    -b $BATCH_SIZE \
+                    -t ${PRECISION} \
+                    -o None 2>&1 | tee ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_c${i}_t${j}_raw.log & ")
+
+            done
+      done
+        
+  #     bertsquad_log_analysis ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0_raw.log ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0.log inference ${BATCH_SIZE}
+  # bertsquad_log_analysis ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t1_raw.log ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t1.log inference ${BATCH_SIZE}
+  str=${str[@]}
+  cmd_line=${str::-2}
+  eval $cmd_line
   wait
-  bertsquad_log_analysis ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0_raw.log ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t0.log inference ${BATCH_SIZE}
-  bertsquad_log_analysis ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t1_raw.log ${OUTPUT_DIR}/bertsquad_fp16_inf_plain_nchw_t1.log inference ${BATCH_SIZE}
   cd -
 else
-    echo "The specified Tile '${Tile}' is unsupported."
-    echo "Supported tile number are: 1 and 2"
+    echo "Currently only x4 OAM Modules are supported"
     exit 1
 fi
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
index d9c1f69ca..c0647f221 100644
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
@@ -140,7 +140,19 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
 ## Quick Start Script for fast_bert with TPP optimization 
 |  DataType   | pre-train  |  finetune|
 | ----------- | ----------- | ----------- |
-| BF16        |bash sh run_fast_bert_pretrain_8_node.sh | bash sh fast_bert_squad_finetune.sh --use_tpp --tpp_bf16 --unpad |
+| BF16        |bash sh run_fast_bert_pretrain.sh | bash sh fast_bert_squad_finetune.sh --use_tpp --tpp_bf16 --unpad |
+
+```
+bash sh run_fast_bert_pretrain.sh
+```
+will run 32 ranks on 8 nodes with `global batch size=2048`.
+You can config following ENV to run more settings, for example
+```
+export NPROCESS=16
+export PROCESS_PER_NODE=2
+export GBS=1024
+```
+will run 16 ranks on 8 nodes with `global batch size=1024`.
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, download and
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/fast_bert_pretrain.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/fast_bert_pretrain.sh
index 73700a537..17b087575 100755
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/fast_bert_pretrain.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/fast_bert_pretrain.sh
@@ -38,6 +38,7 @@ log_event(key=constants.CACHE_CLEAR, value=True)"
 
 
 GBS=2048
+GBS=${GBS:-2048}
 LBS=$(( GBS / NUM_RANKS ))
 
 params="--train_batch_size=${LBS}    --learning_rate=2.0e-3     --opt_lamb_beta_1=0.66     --opt_lamb_beta_2=0.998     --warmup_proportion=0.0     --warmup_steps=0.0     --start_warmup_step=0     --max_steps=1710   --phase2    --max_predictions_per_seq=76      --do_train     --skip_checkpoint     --train_mlm_accuracy_window_size=0     --target_mlm_accuracy=0.720     --weight_decay_rate=0.01     --max_samples_termination=4500000     --eval_iter_start_samples=150000 --eval_iter_samples=150000     --eval_batch_size=16  --gradient_accumulation_steps=1     --log_freq=0 "
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain_8node.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain.sh
similarity index 81%
rename from quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain_8node.sh
rename to quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain.sh
index 859233441..8c1f3556b 100644
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain_8node.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_fast_bert_pretrain.sh
@@ -36,5 +36,8 @@ export I_MPI_DEBUG=5
 #from mlperf_logger import log_event
 #log_event(key=constants.CACHE_CLEAR, value=True)"
 
+NPROCESS=${NPROCESS:-32}
+PROCESS_PER_NODE=${PROCESS_PER_NODE:-4}
+
 export SEED=$(date +%s)
-bash ./run_dist_ht.sh -np 32 -ppn 4 -f hostfile bash fast_bert_pretrain.sh --use_tpp --tpp_bf16 --unpad --dense_seq_out  --dist_lamb  --no_ddp --seed ${SEED} 2>&1|tee output.log
+bash ./run_dist_ht.sh -np $NPROCESS -ppn $PROCESS_PER_NODE -f hostfile bash fast_bert_pretrain.sh --use_tpp --tpp_bf16 --unpad --dense_seq_out  --dist_lamb  --no_ddp --seed ${SEED} 2>&1|tee output.log
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/gpu/bf16_training_plain_format.sh b/quickstart/language_modeling/pytorch/bert_large/training/gpu/bf16_training_plain_format.sh
old mode 100644
new mode 100755
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/gpu/ddp_bf16_training_plain_format.sh b/quickstart/language_modeling/pytorch/bert_large/training/gpu/ddp_bf16_training_plain_format.sh
old mode 100644
new mode 100755
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/training_multinode.sh b/quickstart/language_modeling/pytorch/rnnt/training/cpu/training_multinode.sh
index 72c848b7f..c89c43493 100755
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/training_multinode.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/training/cpu/training_multinode.sh
@@ -133,7 +133,6 @@ python -m intel_extension_for_pytorch.cpu.launch \
     --distributed \
     --nnodes ${NNODES} \
     --hostfile ${HOSTFILE} \
-    --nproc_per_node $SOCKETS \
     --log_path=${OUTPUT_DIR} \
     --log_file_prefix="./distributed_throughput_log_${precision}" \
     ${MODEL_DIR}/models/language_modeling/pytorch/rnnt/training/cpu/train.py \
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/.docs/setup.md b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/.docs/setup.md
deleted file mode 100644
index bea82375d..000000000
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/.docs/setup.md
+++ /dev/null
@@ -1,35 +0,0 @@
-<!-- 20. Environment setup on baremetal -->
-## Setup on baremetal
-
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-
-* Install git, numactl and wget, if not installed already
-  ```bash
-  yum update -y && yum install -y git numactl wget
-  ```
-
-* Install Intel Tensorflow
-  ```bash
-  pip install intel-tensorflow==2.11.dev202242
-  ```
-
-* Install the keras version that works with the above tensorflow version:
-  ```bash
-  pip install keras-nightly==2.11.0.dev2022092907
-  ```
-
-* Note: For kernel version 5.16, AVX512_CORE_AMX is turned on by default. If the kernel version < 5.16 , please set the following environment variable for AMX environment: 
-  ```bash
-  DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
-  # To run VNNI, please set 
-  DNNL_MAX_CPU_ISA=AVX512_CORE_BF16
-  ```
-
-* Clone [Intel Model Zoo repository](https://github.com/IntelAI/models)
-  ```bash
-  git clone https://github.com/IntelAI/models
-  ```
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_SPR_Baremetal.md b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_SPR_Baremetal.md
index 8f54f1328..87da0245a 100644
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_SPR_Baremetal.md
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_SPR_Baremetal.md
@@ -104,8 +104,11 @@ export DATASET_DIR=<path to the dataset>
 export OUTPUT_DIR=<directory where log files will be written>
 export PRETRAINED_MODEL=<path to the downloaded pre-trained model>
 export CHECKPOINT_DIR=<path to the downloaded checkpoints folder>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
+
+#Optional envs
+export BATCH_SIZE=<customized batch size value, otherwise it will run with the default value>
+export OMP_NUM_THREADS=<customized value for omp_num_threads, otherwise it will run with the default value>
+export CORES_PER_INSTANCE=<customized value for cores_per_instance, otherwise it will run with the default value>
 
 Run the script:
 ./quickstart/language_modeling/tensorflow/bert_large/inference/cpu/<script_name.sh>
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh
index 66e7930b0..9d80abdb2 100755
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh
@@ -96,8 +96,19 @@ if [[ $PRECISION == "bfloat32" ]]; then
 fi
 
 MODE="inference"
-CORES_PER_INSTANCE="4"
 
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE="4"
+  echo "Running with default ${CORES_PER_INSTANCE} cores per instance"
+fi
+
+# If OMP_NUM_THREADS env is not mentioned, then run with the default value
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=4
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}" ]; then
   BATCH_SIZE="1"
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh
index 902856e9a..87aa9fda6 100755
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh
@@ -85,7 +85,12 @@ elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
 fi
 
 MODE="inference"
-CORES_PER_INSTANCE="4"
+
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE="4"
+  echo "Running with default ${CORES_PER_INSTANCE} cores per instance"
+fi
 
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}" ]; then
@@ -99,12 +104,16 @@ if [[ $PRECISION == "bfloat32" ]]; then
   PRECISION="fp32"
 fi
 
-export OMP_NUM_THREADS=4
+# If OMP_NUM_THREADS env is not mentioned, then run with the default value
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=4
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 
 # clean up old log files if found
 rm -rf ${OUTPUT_DIR}/Bert_large_${PRECISION}_bs${BATCH_SIZE}_Latency_inference_instance_*
 
-
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
 _get_socket_cores_lists
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh
index 8dae6e8d1..11660fd3e 100755
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh
@@ -86,7 +86,22 @@ elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
 fi
 
 MODE="inference"
-CORES_PER_INSTANCE="socket"
+
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE="socket"
+  echo "Runs an instance per ${CORES_PER_INSTANCE}"
+fi
+
+cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
+cores_per_socket="${cores_per_socket//[[:blank:]]/}"
+
+# If OMP_NUM_THREADS env is not mentioned, then run with the default value
+if [ -z "${OMP_NUM_THREADS}" ]; then 
+  export OMP_NUM_THREADS=${cores_per_socket}
+else
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+fi
 
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}" ]; then
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/profile.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/profile.sh
old mode 100644
new mode 100755
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md
index 4387839f4..495eb590d 100644
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md
@@ -1,7 +1,7 @@
 # TensorFlow BERT Large Inference
 
 ## Description
-This document has instructions for running BERT Large inference with FP16 and FP32 precision using Intel® Data Center GPU Max Series. 
+This document has instructions for running BERT Large inference with FP16,bfloat16 and FP32 precisions using Intel® Data Center GPU Max Series. 
 
 ## Datasets 
 
@@ -23,34 +23,33 @@ Download the SQUAD directory and set the `SQUAD_DIR` environment variable to poi
   wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
   wget https://raw.githubusercontent.com/allenai/bi-att-flow/master/squad/evaluate-v1.1.py
   ```
-
 ## Quick Start Scripts
 | Script name | Description |
 |-------------|-------------|
-| `benchmark.sh` | This script runs bert large fp16 and fp32 inference. |
+| `inference.sh` | This script runs bert large fp16,bfloat16 and fp32 inference. |
 
 ## Docker
 Requirements:
-* Host machine has Intel(R) Data Center Max Series GPU
-* Follow instructions to install GPU-compatible driver [602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
+* Host machine has Intel(R) Data Center Max Series 1550 x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
 * Docker
 
 ## Docker pull Command
 ```
 docker pull intel/language-modeling:tf-max-gpu-bert-large-inference
 ```
-The BERT Large Inference container includes scripts,models,libraries needed to run fp16/fp32 Inference. 
+The BERT Large Inference container includes scripts,models,libraries needed to run fp16,bfloat16 and fp32 Inference. 
 
 ```
-export PRECISION=fp16
+export PRECISION=<provide either fp16,bfloat16 or fp32 precisions>
 export OUTPUT_DIR=<path to output logs>
 export PRETRAINED_DIR=<path to dataset>
 export SQAUD_DIR=<path to squad directory>
-export Tile=2
-
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export INFERENCE_MODE=<provide accuracy,benchmark or profile mode>
 IMAGE_NAME=intel/language-modeling:tf-max-gpu-bert-large-inference
 DOCKER_ARGS="--rm -it"
-SCRIPT=benchmark.sh
+SCRIPT=inference.sh
 
 FROZEN_GRAPH=/workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
 
@@ -61,10 +60,10 @@ docker run \
   --ipc=host \
   --env PRECISION=${PRECISION} \
   --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env FROZEN_GRAPH=${FROZEN_GRAPH} \
   --env PRETRAINED_DIR=${PRETRAINED_DIR} \
   --env SQUAD_DIR=${SQUAD_DIR} \
-  --env Tile=${Tile} \
+  --env NUM_OAM=${NUM_OAM} \
+  --env INFERENCE_MODE=${INFERENCE_MODE} \
   --env http_proxy=${http_proxy} \
   --env https_proxy=${https_proxy} \
   --env no_proxy=${no_proxy} \
@@ -74,4 +73,4 @@ docker run \
   ${DOCKER_ARGS}\
   $IMAGE_NAME \
   /bin/bash quickstart/$SCRIPT
-  ```
\ No newline at end of file
+  ```
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md
index 268b1af83..954a4bd0f 100644
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md
@@ -8,8 +8,14 @@ Intel-optimized TensorFlow with Intel® Data Center GPU Max Series.
 
 <!--- 20. GPU Setup -->
 ## Software Requirements:
-- Intel® Data Center GPU Max Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-tensorflow/latest/get_started.html) to install the latest ITEX version and other prerequisites.
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-tensorflow/) to install the latest ITEX version and other prerequisites.
 
 - Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
   - Intel® oneAPI DPC++ Compiler
@@ -31,8 +37,7 @@ Intel-optimized TensorFlow with Intel® Data Center GPU Max Series.
 ## Datasets
 
 ### BERT Large Data
-Download and unzip the BERT Large uncased (whole word masking) model from the
-[google bert repo](https://github.com/google-research/bert#pre-trained-models).
+Download and unzip the BERT Large uncased (whole word masking) model from the [google bert repo](https://github.com/google-research/bert#pre-trained-models).
 Then, download the Stanford Question Answering Dataset (SQuAD) dataset file `dev-v1.1.json` into the `wwm_uncased_L-24_H-1024_A-16` directory that was just unzipped.
 
 ```
@@ -41,29 +46,28 @@ unzip wwm_uncased_L-24_H-1024_A-16.zip
 
 wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json -P wwm_uncased_L-24_H-1024_A-16
 ```
-Set the `DATASET_DIR` to point to that directory when running BERT Large inference using the SQuAD data.
+Set the `PRETRAINED_DIR` to point to that directory when running BERT Large inference using the SQuAD data.
+
+Download the SQUAD directory and set the `SQUAD_DIR` environment variable to point where it was saved:
+  ```
+  wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
+  wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
+  wget https://raw.githubusercontent.com/allenai/bi-att-flow/master/squad/evaluate-v1.1.py
+  ```
 
 <!--- 40. Quick Start Scripts -->
 ## Quick Start Scripts
 
 | Script name | Description |
 |-------------|-------------|
-| [`benchmark.sh`](benchmark.sh) | This script runs bert large fp16 and fp32 inference. |
-| [`accuracy.sh`](accuracy.sh) | This script runs bert large fp16 and fp32 inference in accuracy mode. |
+| `inference.sh` | This script runs Bert Large inference for precisions (fp16, bfloat16 and fp32). |
 
 
 <!--- 50. Baremetal -->
 ## Run the model
-Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-  
 * Download the frozen graph model file, and set the FROZEN_GRAPH environment variable to point to where it was saved:
   ```bash
-  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/fp32_bert_squad.pb
+  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/bert_frozen_graph_fp32.pb
   ```
 
 * Download the pretrained model directory and set the PRETRAINED_DIR environment variable to point where it was saved:
@@ -88,13 +92,12 @@ Navigate to the BERT Large inference directory, and set environment variables:
 ```
 cd models
 export OUTPUT_DIR=<path where output log files will be written>
-export PRECISION=<Set precision: fp16 or fp32>
+export PRECISION=<provide either fp16,bfloat16 or fp32 precisions>
 export FROZEN_GRAPH=<path to pretrained model file (*.pb)>
 export PRETRAINED_DIR=<path to pretrained directory>
 export SQUAD_DIR=<path to squad directory>
-
-# Set `Tile` env variable only for running `benchmark.sh` script:
-export Tile=2
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export INFERENCE_MODE=<provide accuracy,benchmark or profile mode>
 
 # Run quickstart script:
 ./quickstart/language_modeling/tensorflow/bert_large/inference/gpu/<script name>.sh
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/accuracy.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/accuracy.sh
deleted file mode 100755
index 61c9417e9..000000000
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/accuracy.sh
+++ /dev/null
@@ -1,85 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'PRECISION='$PRECISION
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'PRETRAINED_DIR='$PRETRAINED_DIR
-echo 'SQUAD_DIR='$SQUAD_DIR
-echo 'FROZEN_GRAPH='$FROZEN_GRAPH
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_envs
-input_envs[PRECISION]=${PRECISION}
-input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
-input_envs[PRETRAINED_DIR]=${PRETRAINED_DIR}
-input_envs[SQUAD_DIR]=${SQUAD_DIR}
-input_envs[FROZEN_GRAPH]=${FROZEN_GRAPH}
-
-for i in "${!input_envs[@]}"; do
-  var_name=$i
-  env_param=${input_envs[$i]}
- 
-  if [[ -z $env_param ]]; then
-    echo "The required environment variable $var_name is not set" >&2
-    exit 1
-  fi
-done
-
-# Check for precision
-if [[ ${PRECISION} == "fp16" || ${PRECISION} == "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is supported."
-else
-  echo "The specified precision '${PRECISION}' is not supported. Only fp16 and fp32 precision is supported"
-  exit 1
-fi
-
-lspci_display_info=$(lspci | grep -i display)
-
-if [ -z "${BATCH_SIZE}" ]; then
-      BATCH_SIZE="64"
-      echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-export CreateMultipleSubDevices=1
-export TF_NUM_INTEROP_THREADS=1
-
-if [[ $PRECISION == "fp16" ]]; then
-  export ITEX_AUTO_MIXED_PRECISION=1
-  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-source "quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=${PRECISION} \
-    --mode=inference \
-    --in-graph=${FROZEN_GRAPH} \
-    --framework=tensorflow \
-    --batch-size=${BATCH_SIZE} \
-    --vocab-file=${PRETRAINED_DIR}/vocab.txt \
-    --config-file=${PRETRAINED_DIR}/bert_config.json \
-    --predict-file=${SQUAD_DIR}/dev-v1.1.json \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    --gpu \
-    $@ \
-    -- infer_option=SQuAD
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/benchmark.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/benchmark.sh
deleted file mode 100755
index 4b7eb2214..000000000
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/benchmark.sh
+++ /dev/null
@@ -1,125 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'PRECISION='$PRECISION
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'PRETRAINED_DIR='$PRETRAINED_DIR
-echo 'SQUAD_DIR='$SQUAD_DIR
-echo 'FROZEN_GRAPH='$FROZEN_GRAPH
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_envs
-input_envs[PRECISION]=${PRECISION}
-input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
-input_envs[PRETRAINED_DIR]=${PRETRAINED_DIR}
-input_envs[SQUAD_DIR]=${SQUAD_DIR}
-input_envs[FROZEN_GRAPH]=${FROZEN_GRAPH}
-
-for i in "${!input_envs[@]}"; do
-  var_name=$i
-  env_param=${input_envs[$i]}
- 
-  if [[ -z $env_param ]]; then
-    echo "The required environment variable $var_name is not set" >&2
-    exit 1
-  fi
-done
-
-# Check for precision
-if [[ ${PRECISION} == "fp16" || ${PRECISION} == "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is supported."
-else
-  echo "The specified precision '${PRECISION}' is not supported. Only fp16 and fp32 precision is supported"
-  exit 1
-fi
-
-if [ -z "${BATCH_SIZE}" ]; then
-      BATCH_SIZE="64"
-      echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-if [[ $PRECISION == "fp16" ]]; then
-  export ITEX_AUTO_MIXED_PRECISION=1
-  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
-fi
-
-export CreateMultipleSubDevices=1
-export TF_NUM_INTEROP_THREADS=1
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-source "quickstart/common/utils.sh"
-if [[ ${Tile} == "1" ]]; then
-  echo "Bert-large Inference"
-  _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=${PRECISION} \
-    --mode=inference \
-    --in-graph=${FROZEN_GRAPH} \
-    --framework=tensorflow \
-    --batch-size=${BATCH_SIZE} \
-    --vocab-file=${PRETRAINED_DIR}/vocab.txt \
-    --config-file=${PRETRAINED_DIR}/bert_config.json \
-    --predict-file=${SQUAD_DIR}/dev-v1.1.json \
-    --output-dir ${OUTPUT_DIR} \
-    --max-seq-length=384 \
-    --doc-stride=128 \
-    --benchmark-only \
-    --gpu \
-    $@ \
-    -- infer-option=SQuAD 2>&1 | tee ${OUTPUT_DIR}//bert-large_inf_t0_raw.log
-  
-  elif [[ ${Tile} == "2" ]]; then
-    echo "Bert-large two tile Inference"
-    ZE_AFFINITY_MASK=0.0 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=${PRECISION} \
-    --mode=inference \
-    --in-graph=${FROZEN_GRAPH} \
-    --framework=tensorflow \
-    --batch-size=${BATCH_SIZE} \
-    --vocab-file=${PRETRAINED_DIR}/vocab.txt \
-    --config-file=${PRETRAINED_DIR}/bert_config.json \
-    --predict-file=${SQUAD_DIR}/dev-v1.1.json \
-    --output-dir ${OUTPUT_DIR} \
-    --max-seq-length=384 \
-    --doc-stride=128 \
-    --benchmark-only \
-    --gpu \
-    $@ \
-    -- infer-option=SQuAD 2>&1 | tee ${OUTPUT_DIR}//bert-large_inf_t0_raw.log &
-    ZE_AFFINITY_MASK=0.1 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name=bert_large \
-    --precision=${PRECISION} \
-    --mode=inference \
-    --in-graph=${FROZEN_GRAPH} \
-    --framework=tensorflow \
-    --batch-size=${BATCH_SIZE} \
-    --vocab-file=${PRETRAINED_DIR}/vocab.txt \
-    --config-file=${PRETRAINED_DIR}/bert_config.json \
-    --predict-file=${SQUAD_DIR}/dev-v1.1.json \
-    --output-dir ${OUTPUT_DIR} \
-    --max-seq-length=384 \
-    --doc-stride=128 \
-    --benchmark-only \
-    --gpu \
-    $@ \
-    -- infer-option=SQuAD 2>&1 | tee ${OUTPUT_DIR}//bert-large_inf_t1_raw.log
-fi
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/build.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/build.sh
deleted file mode 100755
index 2b99231a8..000000000
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/build.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-TENSORFLOW_BASE_IMAGE=${TENSORFLOW_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-TENSORFLOW_BASE_TAG=${TENSORFLOW_BASE_TAG:-gpu-max}
-IMAGE_NAME=${IMAGE_NAME:-intel/language-modeling:tf-max-gpu-bert-large-inference}
-
-docker build \
-    --build-arg PACKAGE_DIR=model_packages \
-    --build-arg PACKAGE_NAME=tf-max-series-bert-large-inference \
-    --build-arg MODEL_WORKSPACE=/workspace \
-    --build-arg http_proxy=$http_proxy \
-    --build-arg https_proxy=$https_proxy \
-    --build-arg no_proxy=$no_proxy \
-    --build-arg TENSORFLOW_BASE_IMAGE=${TENSORFLOW_BASE_IMAGE} \
-    --build-arg TENSORFLOW_BASE_TAG=${TENSORFLOW_BASE_TAG} \
-    -t $IMAGE_NAME \
-    -f tf-max-series-bert-large-inference.Dockerfile .
-
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/inference.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/inference.sh
new file mode 100755
index 000000000..4b63090af
--- /dev/null
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/inference.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2020 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+
+echo 'MODEL_DIR='$MODEL_DIR
+echo 'PRECISION='$PRECISION
+echo 'OUTPUT_DIR='$OUTPUT_DIR
+echo 'PRETRAINED_DIR='$PRETRAINED_DIR
+echo 'SQUAD_DIR='$SQUAD_DIR
+echo 'FROZEN_GRAPH='$FROZEN_GRAPH
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[PRETRAINED_DIR]=${PRETRAINED_DIR}
+input_envs[SQUAD_DIR]=${SQUAD_DIR}
+input_envs[FROZEN_GRAPH]=${FROZEN_GRAPH}
+input_envs[NUM_OAM]=${NUM_OAM}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+# Check for precision
+if [[ ${PRECISION} == "fp16" || ${PRECISION} == "bfloat16" || ${PRECISION} == "fp32" ]]; then
+  echo "The specified precision '${PRECISION}' is supported."
+else
+  echo "The specified precision '${PRECISION}' is not supported. Only fp16 and fp32 precision is supported"
+  exit 1
+fi
+
+if [ -z "${BATCH_SIZE}" ]; then
+  BATCH_SIZE="64"
+  echo "Running with default batch size of ${BATCH_SIZE}"
+fi
+
+if [[ ! -f "${FROZEN_GRAPH}" ]]; then
+  frozen_graph=/workspace/tf-max-series-bert-large-inference/frozen_graph/fp32_bert_squad.pb
+else
+  frozen_graph=${FROZEN_GRAPH}
+fi
+
+if [[ ! -f "${frozen_graph}" ]]; then
+  echo "Please set the path for frozen graph"
+fi
+
+if [[ $PRECISION == "fp16" ]]; then
+  export ITEX_AUTO_MIXED_PRECISION=1
+  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
+fi
+
+if [[ $PRECISION == "bfloat16" ]]; then
+  export ITEX_AUTO_MIXED_PRECISION=1
+  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="BFLOAT16"
+fi
+
+export CreateMultipleSubDevices=1
+export TF_NUM_INTEROP_THREADS=1
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+declare -a str
+
+if [[ ${NUM_OAM} == "4" ]]; then
+  echo "Bert-large Inference on ${NUM_OAM} OAM modules"
+  NUM_TILES_PER_GPU=2
+  cd ${MODEL_DIR}/models/language_modeling/tensorflow/bert_large/inference/gpu
+  for i in $( eval echo {0..$((NUM_OAM-1))} )
+    do
+        for j in $( eval echo {0..$((NUM_TILES_PER_GPU-1))} )
+            do
+            str+=("ZE_AFFINITY_MASK="${i}"."${j}" python -u run_squad.py \
+                  --vocab_file=${PRETRAINED_DIR}/vocab.txt \
+                  --bert_config_file=${PRETRAINED_DIR}/bert_config.json \
+                  --do_train=False \
+                  --train_file=${SQUAD_DIR}/train-v1.1.json \
+                  --do_predict=True \
+                  --predict_file=${SQUAD_DIR}/dev-v1.1.json \
+                  --predict_batch_size=${BATCH_SIZE} \
+                  --learning_rate=3e-5 \
+                  --num_train_epochs=2.0 \
+                  --max_seq_length=384 \
+                  --doc_stride=128 \
+                  --precision=${PRECISION} \
+                  --output_dir=${OUTPUT_DIR} \
+                  --input_graph=${frozen_graph} \
+                  --mode=${INFERENCE_MODE} 2>&1 | tee ${OUTPUT_DIR}/bertsquad_${PRECISION}_${INFERENCE_MODE}_c${i}_t${j}_raw.log & ")
+            done
+    done
+  str=${str[@]}
+  cmd_line=${str::-2}
+  eval $cmd_line
+else 
+    echo "Currently only x4 OAM Modules are supported"
+    exit 1
+fi
diff --git a/quickstart/language_modeling/tensorflow/bert_large/training/gpu/setup.sh b/quickstart/language_modeling/tensorflow/bert_large/training/gpu/setup.sh
old mode 100644
new mode 100755
diff --git a/quickstart/language_translation/tensorflow/bert/inference/cpu/.docs/aikit.md b/quickstart/language_translation/tensorflow/bert/inference/cpu/.docs/aikit.md
deleted file mode 100644
index 8bd85b6c8..000000000
--- a/quickstart/language_translation/tensorflow/bert/inference/cpu/.docs/aikit.md
+++ /dev/null
@@ -1,70 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-* Download the fp32 "BERT-Base, Uncased" pre-trained model and unzip it inside the dataset directory `data/MRPC`.
-If you run on Windows, please use a browser to download the pretrained model using the link below. For Linux, run:
-
-   ```
-   cd data/MRPC
-   wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
-   unzip uncased_L-12_H-768_A-12.zip
-   cd ../..
-   ```
-
-* Clone [google-research/bert](https://github.com/google-research/bert) repository:
-   ```
-   git clone --single-branch https://github.com/google-research/bert.git && cd bert/
-   git checkout 88a817c37f788702a363ff935fd173b6dc6ac0d6
-   ```
-   Set the `MODEL_SOURCE` environment variable to the location of `bert` directory.
-
-* Small sets like MRPC have a high variance in the Dev set accuracy, even when starting from the same pre-training checkpoint.
-  If you re-run multiple times (making sure to point to different `output-dir`), you should see results between 84% and 88%.
-
-
-* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
-  ```
-  git clone https://github.com/IntelAI/models.git
-  ```
-
-* Install [intel-tensorflow>=2.5.0](https://pypi.org/project/intel-tensorflow/) on your system.
-
-
-## Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-# Set env vars
-export DATASET_DIR=<path to the MRPC directory>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model which is downloaded under MRPC directory>
-export MODEL_SOURCE=<path to the bert directory>
-
-# Run a quickstart script
-./quickstart/language_translation/tensorflow/bert/inference/cpu/<script name>.sh
-```
-
-## Run on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-cd models
-
-# Set environment variables
-set DATASET_DIR=<path to the MRPC directory>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-set PRETRAINED_MODEL=<path to the pretrained model which is downloaded under MRPC directory>
-set MODEL_SOURCE=<path to the bert directory>
-
-# Run a quickstart script
-bash quickstart\language_translation\tensorflow\bert\inference\cpu\<script name>.sh
-```
-
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables.
-As an example, if the pretrained model path on Windows is `D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb
-> /d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb
->```
->Then, set the `PRETRAINED_MODEL` environment variable `set PRETRAINED_MODEL=/d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb`.
diff --git a/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh b/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh
index dc2c77a00..8e6f6515a 100755
--- a/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh
+++ b/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh
@@ -49,15 +49,20 @@ fi
 cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
 cores_per_socket="${cores_per_socket//[[:blank:]]/}"
 
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+CORES_PER_NUMA=`expr $CORES \* $SOCKETS / $NUMAS`
+
 # Select the num_intra_threads
 if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat32" ]; then
-  num_intra_threads=$(($cores_per_socket - 4))
+  num_intra_threads=$(($CORES_PER_NUMA - 4))
 elif [ ${PRECISION} == "bfloat16" ]; then
   export KMP_BLOCKTIME=5
-  num_intra_threads=$(($cores_per_socket - 2))
+  num_intra_threads=$(($CORES_PER_NUMA - 2))
 fi
 
-NUM_INSTANCES="1"
+NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
 
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}"]; then
@@ -88,7 +93,7 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --data-location=${DATASET_DIR} \
   --output-dir ${OUTPUT_DIR} \
   --mpi_num_processes=${NUM_INSTANCES} \
-  --mpi_num_processes_per_socket=1 \
+  --mpi_num_processes_per_socket=${NUM_INSTANCES} \
   --batch-size ${BATCH_SIZE} \
   --num-intra-threads ${num_intra_threads} \
   --num-inter-threads 2 \
diff --git a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training_multinode.sh b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training_multinode.sh
index 1aaafe167..1b73e3716 100755
--- a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training_multinode.sh
+++ b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training_multinode.sh
@@ -80,15 +80,14 @@ PRECISION=$1
 oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
 source $oneccl_bindings_for_pytorch_path/env/setvars.sh
 
-rm -rf ${OUTPUT_DIR}/distributed_throughput_log_${PRECISION}*
+rm -rf ${OUTPUT_DIR}/maskrcnn_dist_training_log_${PRECISION}*
 
 python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator tcmalloc \
     --distributed \
     --nnodes ${NNODES} \
     --hostfile ${HOSTFILE} \
-    --nproc_per_node $SOCKETS \
-    --log_path=${OUTPUT_DIR} \
-    --log_file_prefix="./distributed_throughput_log_${PRECISION}" \
+    --logical-cores-for-ccl --ccl_worker_count 8 \
     ${MODEL_DIR}/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/tools/train_net.py \
     $ARGS \
     --iter-warmup 10 \
@@ -101,11 +100,11 @@ python -m intel_extension_for_pytorch.cpu.launch \
     SOLVER.STEPS '"(60000, 80000)"' \
     SOLVER.BASE_LR 0.0025 \
     MODEL.DEVICE cpu \
-    2>&1 | tee ${OUTPUT_DIR}/distributed_throughput_log_${PRECISION}.txt
+    2>&1 | tee ${OUTPUT_DIR}/maskrcnn_dist_training_log_${PRECISION}.log
 
 # For the summary of results
 wait
-throughput=$(grep 'Training throughput:' ${OUTPUT_DIR}/distributed_throughput_log_${PRECISION}* |sed -e 's/.*Training throughput//;s/[^0-9.]//g' |awk '
+throughput=$(grep 'Training throughput:' ${OUTPUT_DIR}/maskrcnn_dist_training_log_${PRECISION}.log |sed -e 's/.*Training throughput//;s/[^0-9.]//g' |awk '
 BEGIN {
         sum = 0;
         i = 0;
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
index 9ee191df3..0993e4326 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
@@ -42,8 +42,17 @@ if [[ "$1" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+
+BATCH_SIZE=112
+
 ARGS=""
 if [[ "$1" == "int8" || "$1" == "avx-int8" ]]; then
+    NUMA_NODES=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+    CORES_PER_NODE=`expr $TOTAL_CORES / $NUMA_NODES`
+    BATCH_SIZE=`expr $CORES_PER_NODE \* 2`
     ARGS="$ARGS --int8"
     ARGS="$ARGS --seed 1 --threshold 0.2 --configure ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/inference/cpu/pytorch_default_recipe_ssd_configure.json"
     export DNNL_GRAPH_CONSTANT_CACHE=1
@@ -124,7 +133,6 @@ if [ "$weight_sharing" = true ]; then
     wait
 
 else
-    BATCH_SIZE=112
     python -m intel_extension_for_pytorch.cpu.launch \
         --throughput_mode \
         ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/inference/cpu/infer.py \
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy_dist.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy_dist.sh
index 2f1dd4f2d..d5bc4489e 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy_dist.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy_dist.sh
@@ -81,11 +81,9 @@ source $oneccl_bindings_for_pytorch_path/env/setvars.sh
 
 python -m intel_extension_for_pytorch.cpu.launch \
     --memory-allocator jemalloc \
-    --ncore_per_instance ${CORES_PER_INSTANCE} \
     --distributed \
     --nnodes ${NNODES} \
     --hostfile ${HOSTFILE} \
-    --nproc_per_node ${SOCKETS} \
     --logical_core_for_ccl --ccl_worker_count 8 \
     ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py \
     --epochs 5 \
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
index 758da9cb8..b7e8dc78d 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
@@ -81,11 +81,9 @@ source $oneccl_bindings_for_pytorch_path/env/setvars.sh
 
 python -m intel_extension_for_pytorch.cpu.launch \
     --memory-allocator jemalloc \
-    --ncore_per_instance ${CORES_PER_INSTANCE} \
     --distributed \
     --nnodes ${NNODES} \
     --hostfile ${HOSTFILE} \
-    --nproc_per_node ${SOCKETS} \
     --logical_core_for_ccl --ccl_worker_count 8 \
     ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/training/cpu/train.py \
     --epochs 70 \
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/baremetal.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/baremetal.md
deleted file mode 100644
index 0ff4ad99c..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/baremetal.md
+++ /dev/null
@@ -1,46 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-Install the following pre-requisites:
-* Python version 3.9
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Install TensorFlow and Intel® Extension for TensorFlow (ITEX):
-
-  Intel® Extension for TensorFlow requires stock TensorFlow v2.10.0 to be installed 
-  
-  ```bash
-  pip install tensorflow==2.10.0
-  pip install --upgrade intel-extension-for-tensorflow[gpu]
-  ```
-   To verify that TensorFlow and ITEX are correctly installed:
-  ```
-  python -c "import intel_extension_for_tensorflow as itex; print(itex.__version__)"
-  ```
-* Download the frozen graph model file, and set the FROZEN_GRAPH environment variable to point to where it was saved:
-  ```bash
-  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/ssd_mobilenet_v1_int8_itex.pb
-  ```
-* Install model specific dependencies:
-  ```bash
-  pip install pycocotools
-  ```
-
-See the [datasets section](#datasets) of this document for instructions on
-downloading and preprocessing the ImageNet dataset. The path to the ImageNet
-TF records files will need to be set as the `DATASET_DIR` environment variable
-prior to running a [quickstart script](#quick-start-scripts).
-
-### To run the model on Baremetal
-This snippet shows how to run a quickstart script:
-```
-export DATASET_DIR=<path to the preprocessed COCO TF dataset>
-export OUTPUT_DIR=<path to where output log files will be written>
-export PRECISION=int8
-export FROZEN_GRAPH=<path to pretrained model file (*.pb)>
-
-Run quickstart script:
-./quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/<script name>.sh
-```
\ No newline at end of file
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/description.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/description.md
deleted file mode 100644
index 28dba6026..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel(R) Extension for TensorFlow with <device>.
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/docker.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/docker.md
deleted file mode 100644
index 11e42df15..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/docker.md
+++ /dev/null
@@ -1,46 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-Requirements:
-* Host machine has Intel(R) Data Center GPU Flex Series.
-* GPU-compatible drivers need to be installed. Please follow the [link](https://registrationcenter.intel.com/en/products/download/4125/) to download.
-* Docker
-* Download and build the [Intel® Extension for TensorFlow (ITEX) container](https://registrationcenter.intel.com/en/products/subscription/956/)(`model-zoo:tensorflow-itex-gpu`)
-
-After extracting the <package name>, use the `build.sh`
-script to build the container. After the container has been built, you can
-run the model <mode> using the `run.sh` script. Set environment variables
-for DATASET_DIR to the path to COCO TF records files,if you do not set it 
-dummy data will be used. However, for accuracy you have to provide the real dataset.
-Set OUTPUT_DIR to an output directory where log files will be written. 
-
-The `run.sh` script will execute one of the [quickstart](#quickstart) script
-using the container that was just built. By default, the
-`quickstart/online_inference.sh` script will be run. To change which
-script gets run, either edit the `run.sh` file, or specify the name of file
-to run using the `SCRIPT` environment variable.
-See the snippet below for an example on how to do this.
-
-> Note: Ensure that your system has the proxy environment variables
-> set, otherwise the container build may fail when trying to pull external
-> dependencies (like apt-get and pip installs).
-
-```
-#Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-
-#Set environment variables
-export PRECISION=int8
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the container with the default online_inference.sh script
-./run.sh
-
-# Specify a different quickstart script to run
-SCRIPT=quickstart/batch_inference.sh ./run.sh
-```
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md
deleted file mode 100644
index f3311e42e..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md
+++ /dev/null
@@ -1,29 +0,0 @@
-<!--- 20. GPU Setup -->
-## Hardware Requirements:
-- Intel® Data Center GPU Flex Series
-
-## Software Requirements:
-- Ubuntu 20.04 (64-bit)
-- Intel GPU Drivers: Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html)
-
-  |Release|OS|Intel GPU|Install Intel GPU Driver|
-    |-|-|-|-|
-    |v1.0.0|Ubuntu 20.04|Intel® Data Center GPU Flex Series| Refer to the [Installation Guides](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-focal-dc.html) for latest driver installation. If install the verified Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html), please append the specific version after components, such as `apt-get install intel-opencl-icd=22.28.23726.1+i419~u20.04`|
-
-- Intel® oneAPI Base Toolkit 2022.3: Need to install components of Intel® oneAPI Base Toolkit
-  - Intel® oneAPI DPC++ Compiler
-  - Intel® oneAPI Math Kernel Library (oneMKL)
-  * Download and install the verified DPC++ compiler and oneMKL in Ubuntu 20.04.
-
-    ```bash
-    $ wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18852/l_BaseKit_p_2022.3.0.8767_offline.sh
-    # 4 components are necessary: DPC++/C++ Compiler, DPC++ Libiary, Threading Building Blocks and oneMKL
-    $ sh ./l_BaseKit_p_2022.3.0.8767_offline.sh
-    ```
-    For any more details, please follow the procedure in https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html.
-
-  - Set environment variables
-    Default installation location {ONEAPI_ROOT} is /opt/intel/oneapi for root account, ${HOME}/intel/oneapi for other accounts
-    ```bash
-    source {ONEAPI_ROOT}/setvars.sh
-    ```
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/package.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/package.md
deleted file mode 100644
index 07471ed4a..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/package.md
+++ /dev/null
@@ -1,26 +0,0 @@
-<!--- 20. Model package -->
-## Model Package
-
-The model package includes the scripts and libraries needed to
-build and run <model name> <mode> using a docker container. Note that
-this model container uses the Tensorflow ITEX GPU container as it's base,
-and it requires the `model-zoo:tensorflow-itex-gpu` image to be built before
-the model container is built.
-```
-<package dir>
-├── build.sh
-├── info.txt
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-│       ├── Intel_Model_Zoo_v2.0_Container_tpps.txt
-│       ├── Intel_Model_Zoo_v2.0_ML_Container_tpps.txt
-│       ├── Intel_Model_Zoo_v2.3_PyTorch.txt
-│       └── licenses.txt
-├── model_packages
-│   └── <package name>
-├── README.md
-├── run.sh
-└── <package dir>.Dockerfile
-```
-
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/quickstart.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/quickstart.md
deleted file mode 100644
index 48fec10a8..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|:-------------:|:-------------:|
-| [`online_inference.sh`](online_inference.sh) | Runs online inference for int8 precision |
-| [`batch_inference.sh`](batch_inference.sh) | Runs batch inference for int8 precision |
-| [`accuracy.sh`](accuracy.sh) | Measures the model accuracy for int8 precision  |
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/title.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/setup_spr.sh b/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/setup_spr.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/setup_spr.sh b/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/setup_spr.sh
old mode 100644
new mode 100755
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh b/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh
index 2bd948271..bd4eeb57b 100755
--- a/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh
+++ b/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh
@@ -69,7 +69,12 @@ cd ${MODEL_DIR}
 cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
 cores_per_socket="${cores_per_socket//[[:blank:]]/}"
 
-NUM_INSTANCES="1"
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+CORES_PER_NUMA=`expr $CORES \* $SOCKETS / $NUMAS`
+
+NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
 
 #Set up env variable for bfloat32
 if [[ $PRECISION == "bfloat32" ]]; then
@@ -94,9 +99,9 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --output-dir ${OUTPUT_DIR} \
   --model-source-dir ${TF_MODELS_DIR} \
   --mpi_num_processes=${NUM_INSTANCES} \
-  --mpi_num_processes_per_socket=1 \
+  --mpi_num_processes_per_socket=${NUM_INSTANCES} \
   --batch-size ${BATCH_SIZE} \
-  --num-intra-threads ${cores_per_socket} \
+  --num-intra-threads ${CORES_PER_NUMA} \
   --num-inter-threads 1 \
   --num-cores ${cores_per_socket} \
   --synthetic-data --num-train-steps 100 --num_warmup_batches=20 --weight_decay=1e-4 \
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh b/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
index a13b2d831..25275184a 100755
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
@@ -1,3 +1,4 @@
+#!/bin/bash
 # Copyright (c) 2021 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh b/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
index 0ed93d6e2..3166b695b 100755
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
@@ -1,3 +1,4 @@
+#!/bin/bash
 # Copyright (c) 2021 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -47,11 +48,16 @@ LOG=${OUTPUT_DIR}/dlrm_inference_performance_log/${PRECISION}
 rm -rf ${LOG}
 mkdir -p ${LOG}
 
-CORES=`lscpu | grep Core | awk '{print $4}'`
+CORES_PER_SOCKET=`lscpu | grep "Core(s) per socket" | awk '{print $4}'`
+SOCKETS=`lscpu | grep "Socket(s)" | awk '{print $2}'`
+NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
+NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
+CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
+
 ARGS=""
 if [[ $PRECISION == "int8" ]]; then
     echo "running int8 path"
-    ARGS="$ARGS --num-cpu-cores=$CORES --int8 --int8-configure=${MODEL_DIR}/models/recommendation/pytorch/dlrm/product/int8_configure.json"
+    ARGS="$ARGS --num-cpu-cores=$CORES_PER_NUMA_NODE --int8 --int8-configure=${MODEL_DIR}/models/recommendation/pytorch/dlrm/product/int8_configure.json"
 elif [[ $PRECISION == "bf16" ]]; then
     ARGS="$ARGS --bf16"
     echo "running bf16 path"
@@ -67,7 +73,6 @@ else
 fi
 
 export OMP_NUM_THREADS=1
-
 LOG_0="${LOG}/throughput.log"
 python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc $MODEL_SCRIPT \
 --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
@@ -76,7 +81,7 @@ python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jema
 --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
 --arch-sparse-feature-size=128 --max-ind-range=40000000 --ipex-interaction \
 --numpy-rand-seed=727  --inference-only --num-batches=1000 \
---print-freq=10 --print-time --test-mini-batch-size=128 --share-weight-instance=$CORES \
+--print-freq=10 --print-time --test-mini-batch-size=128 --share-weight-instance=$CORES_PER_NUMA_NODE \
 $ARGS |tee $LOG_0
 wait
 
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
index 0f2b49855..3f9623253 100644
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
@@ -96,6 +96,14 @@ bash test_convergence.sh
 
 # Run quickstart to distribute training dlrm on 2 sockets
 # Note, you need to follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Torch-CCL and run this command on the machine which sockets larger than 2
+# The default setting will run 2 ranks on 1 nodes with BATCH_SIZE=32768 and CCL_WORKER_COUNT=8
+NUM_BATCH=10000 bash distribute_training.sh
+
+# To run more settings, you can config following ENV
+export BATCH_SIZE=65536
+export NUM_CCL_WORKER=4
+export HOSTFILE=<your hostfile>
+export NODE=2
 NUM_BATCH=10000 bash distribute_training.sh
 ```
 
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh b/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
index d9b208d08..50f29e5c6 100644
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
@@ -72,29 +72,38 @@ else
     exit 1
 fi
 
+BATCHSIZE=${BATCHSIZE:-32768}
+NUM_CCL_WORKER=${NUM_CCL_WORKER:-8}
+HOSTFILE=${HOSTFILE:-hostfile1}
 
-CORES=`lscpu | grep Core | awk '{print $4}'`
-SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-BATCHSIZE=32768
 seed_num=1665468325 #1665462256 #$(date +%s)
 oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
 source $oneccl_bindings_for_pytorch_path/env/setvars.sh
-export CCL_MNIC=local
-export CCL_MNIC_COUNT=2
-export FI_PROVIDER=psm3
+export TORCH_CPP_LOG_LEVEL=INFO
+export TORCH_DISTRIBUTED_DEBUG=INFO
+export CCL_LOG_LEVEL=info
+# export CCL_MNIC=local
+# export FI_PROVIDER=psm3
+# export CCL_ATL_TRANSPORT=ofi
+
 export CCL_ALLREDUCE=rabenseifner
-export PSM3_IDENTIFY=1
-export PSM3_ALLOW_ROUTERS=1
-export PSM3_RDMA=1 
-export PSM3_RV_MR_CACHE_SIZE=8192 
-export FI_PROVIDER_PATH=/usr/lib64/libfabric
-export CCL_MNIC_NAME=irdma-cvl01tf2,irdma-cvl02tf2,irdma-cvl11tf2,irdma-cvl12tf2
-export CCL_MNIC_COUNT=2
+# export PSM3_IDENTIFY=1
+# export PSM3_ALLOW_ROUTERS=1
+# export PSM3_RDMA=1 
+# export PSM3_RV_MR_CACHE_SIZE=8192 
+# export FI_PROVIDER_PATH=/usr/lib64/libfabric
+# export CCL_MNIC_NAME=irdma-cvl01tf2,irdma-cvl02tf2,irdma-cvl11tf2,irdma-cvl12tf2
+# export CCL_MNIC_COUNT=2
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+CORES_PER_INSTANCE=$CORES
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+
 LOCAL_BATCH_SIZE=$((BATCHSIZE / NODE))
 LOCAL_BATCH_SIZE=$((LOCAL_BATCH_SIZE / 2))
 
 LOG_0="${LOG}/socket.log"
-python -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --logical_core_for_ccl --ccl_worker_count 8 --distributed --hostfile hostfile1 --nnodes $NODE \
+python -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --logical_core_for_ccl --ccl_worker_count $NUM_CCL_WORKER --distributed --hostfile $HOSTFILE --nnodes $NODE \
 $MODEL_SCRIPT \
   --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
   --data-set=terabyte \
@@ -103,7 +112,7 @@ $MODEL_SCRIPT \
   --arch-sparse-feature-size=128 --max-ind-range=40000000 \
   --numpy-rand-seed=${seed_num} --print-auc --mlperf-auc-threshold=0.8025 \
   --lr-num-warmup-steps=8000   --lr-decay-start-step=70000 --lr-num-decay-steps=30000\
-  --local-batch-size=${LOCAL_BATCH_SIZE} --print-freq=1024 --print-time --ipex-interaction \
+  --local-batch-size=${LOCAL_BATCH_SIZE} --print-freq=100 --print-time --ipex-interaction \
   --test-mini-batch-size=65536 --ipex-merged-emb --should-test --test-freq 6400\
   $ARGS |tee $LOG_0
 wait
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh b/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
old mode 100644
new mode 100755
index f2b873d5b..64fba8392
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
@@ -1,3 +1,4 @@
+#!/bin/bash
 # Copyright (c) 2021 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/.gitignore b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/.gitignore
new file mode 100644
index 000000000..00aae7443
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/.gitignore
@@ -0,0 +1 @@
+memrec_out
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md
new file mode 100644
index 000000000..53287964b
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md
@@ -0,0 +1,96 @@
+<!--- 0. Title -->
+# PyTorch MEMREC-DLRM inference
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running MEMREC-DLRM inference using
+Intel-optimized PyTorch for bare metal.
+
+## Bare Metal
+Setting up MEMREC-DLRM:
+-----------------
+- Create conda environment: ```conda create -n memrec python=3.11```
+- Activate conda environment: ```conda activate memrec```
+- Install Intel® Extension for PyTorch:
+    ```pip install intel-extension-for-pytorch```
+- Install torch: 
+  - CPU: ```conda install pytorch torchvision torchaudio cpuonly -c pytorch```
+- Install mlperf logger:
+  - ```git clone https://github.com/mlperf/logging.git mlperf-logging```
+  - ```pip install -e mlperf-logging```
+- Install dependencies: ```pip install -r quickstart/recommendation/pytorch/memrec_dlrm/requirements.txt```
+- Install memrec encoder:
+  - ```cd models/recommendation/pytorch/memrec_dlrm/memrec_encoder/code/```
+  - ```python setup.py install --user```
+
+## Datasets
+
+### Criteo Terabyte Dataset
+
+The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is
+used to run MEMREC. To download the dataset, you will need to visit the Criteo website and accept
+their terms of use:
+[https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/).
+Copy the download URL into the command below as the `<download url>` and
+replace the `<dir/to/save/MEMREC_data>` to any path where you want to download
+and save the dataset.
+```bash
+export DATASET_DIR=<dir/to/save/MEMREC_data>
+
+mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
+curl -O <download url>/day_{$(seq -s , 0 23)}.gz
+gunzip day_*.gz
+```
+The raw data will be automatically preprocessed and saved as `day_*.npz` to
+the `DATASET_DIR` when MEMREC is run for the first time. On subsequent runs, the
+scripts will automatically use the preprocessed data.
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32). |
+| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32). |
+
+## Run the model
+
+Download the [MEMREC-DLRM PyTorch weights](https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/memrec_CriteoTB_D75000dw75000_K1kw1.pt)
+and set the `WEIGHT_PATH` to point to the weights file.
+```
+export WEIGHT_PATH=<path to the memrec_CriteoTB_D75000dw75000_K1kw1.pt file>
+```
+
+Follow the instructions above to setup your bare metal environment, do the
+model-specific setup and download and prepropcess the dataset. Once all the
+setup is done, the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+Ensure that you have enviornment variables set to point to the dataset directory,
+precision, weights file, and an output directory.
+
+```bash
+# Clone the model zoo repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+
+# Env vars
+export PRECISION=<specify the precision to run>
+export WEIGHT_PATH=<path to the memrec pt file> # only needed for testing accuracy
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+
+# Run a quickstart script (for example, bare metal performance)
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu
+bash inference_performance.sh
+
+# Evaluate Inference Accuracy (for example, bare metal performance)
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu
+bash accuracy.sh
+```
+
+<!--- 80. License -->
+## License
+
+[LICENSE](../../../../../../models/recommendation/pytorch/memrec_dlrm/memrec_encoder/LICENSE)
+
+[THIRD PARTY LICENSES](../../../../../../models/recommendation/pytorch/memrec_dlrm/memrec_encoder/THIRD-PARTY-LICENSES)
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/accuracy.sh b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/accuracy.sh
new file mode 100755
index 000000000..c1b40e5fe
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/accuracy.sh
@@ -0,0 +1,87 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/memrec_dlrm/product/memrec.py"  ]; then
+    echo "Could not find the script of memrec.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the memrec.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/memrec_dlrm/product/memrec.py"
+    exit 1
+fi
+
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/memrec_dlrm/product/memrec.py
+
+echo "PRECISION: ${PRECISION}"
+echo "DATASET_DIR: ${DATASET_DIR}"
+echo "WEIGHT_PATH: ${WEIGHT_PATH}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+  exit 1
+fi
+
+if [ -z "${WEIGHT_PATH}" ]; then
+  echo "The required environment variable WEIGHT_PATH has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}/memrec_inference_accuracy_log
+
+LOG=${OUTPUT_DIR}/memrec_inference_accuracy_log/${PRECISION}
+
+if [[ "$PRECISION" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+ARGS=""
+if [[ $PRECISION == "fp32" ]]; then
+    echo "running fp32 path"
+else
+    echo "The specified PRECISION '${PRECISION}' is unsupported."
+    echo "Supported PRECISIONs are: fp32"
+    exit 1
+fi
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+# use first socket
+numa_cmd="numactl -C 0-$((CORES-1))  "
+echo "will run on core 0-$((CORES-1)) on socket 0" 
+
+export OMP_NUM_THREADS=$CORES
+python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
+--raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+--data-set=terabyte \
+--memory-map --mlperf-bin-loader --round-targets=True \
+--arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+--arch-sparse-feature-size=128 --max-ind-range=40000000 \
+--numpy-rand-seed=727  --inference-only \
+--print-freq=1000 --print-time --mini-batch-size=2048 --test-mini-batch-size=16384 \
+--test-freq=2048 --print-auc $ARGS \
+--learning-rate=0.50  --D 75000 --K 1 --dw 75000 --kw 1 --arch-mlp-bot-sparse 128 \
+--load-model=${WEIGHT_PATH} | tee $LOG
+
+accuracy=$(grep 'Accuracy:' $LOG |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
+echo ""memrec";"auc";${PRECISION};16384;${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/inference_performance.sh b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/inference_performance.sh
new file mode 100755
index 000000000..33ec1e03d
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/inference_performance.sh
@@ -0,0 +1,94 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/memrec_dlrm/product/memrec.py"  ]; then
+    echo "Could not find the script of memrec.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the memrec.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/dlrm/product/memrec.py"
+    exit 1
+fi
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/memrec_dlrm/product/memrec.py
+
+echo "PRECISION: ${PRECISION}"
+echo "DATASET_DIR: ${DATASET_DIR}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+  exit 1
+fi
+
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+LOG=${OUTPUT_DIR}/memrec_inference_performance_log/${PRECISION}
+rm -rf ${LOG}
+mkdir -p ${LOG}
+
+ARGS=""
+if [[ $PRECISION == "fp32" ]]; then
+    echo "running fp32 path"
+else
+    echo "The specified PRECISION '${PRECISION}' is unsupported."
+    echo "Supported PRECISIONs are: fp32"
+    exit 1
+fi
+
+export OMP_NUM_THREADS=1
+CORES_PER_SOCKET=`lscpu | grep "Core(s) per socket" | awk '{print $4}'`
+SOCKETS=`lscpu | grep "Socket(s)" | awk '{print $2}'`
+NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
+NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
+CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
+
+LOG_0="${LOG}/throughput.log"
+
+python3 -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc $MODEL_SCRIPT \
+--raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+--data-set=terabyte \
+--memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+--arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+--arch-sparse-feature-size=128 --max-ind-range=40000000  \
+--numpy-rand-seed=727  --inference-only --num-batches=1000 \
+--print-freq=10 --print-time --test-mini-batch-size=128 --share-weight-instance=$CORES_PER_NUMA_NODE \
+--learning-rate=0.50  --D 75000 --K 1 --dw 75000 --kw 1 --arch-mlp-bot-sparse 128 \
+
+$ARGS |tee $LOG_0
+wait
+
+throughput=$(grep 'Throughput:' ${LOG}/throughput.log |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
+BEGIN {
+        sum = 0;
+        i = 0;
+      }
+      {
+        sum = sum + $1;
+        i++;
+      }
+END   {
+sum = sum / i;
+        printf("%.3f", sum);
+}')
+echo ""dlrm";"throughput";${PRECISION};128;${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run.sh b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run.sh
new file mode 100755
index 000000000..474c8eb5d
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  exit 1
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${WEIGHT_PATH}" ]; then
+  echo "The required environment variable WEIGHT_PATH has not been set"
+  exit 1
+fi
+
+IMAGE_NAME=${IMAGE_NAME:-model-zoo:pytorch-spr-dlrm-inference}
+DOCKER_ARGS=${DOCKER_ARGS:---privileged --init -it}
+WORKDIR=/workspace/pytorch-spr-dlrm-inference
+
+# inference scripts:
+# inference_performance.sh
+# accuracy.sh
+export SCRIPT="${SCRIPT:-inference_performance.sh}"
+
+if [[ ${SCRIPT} != quickstart* ]]; then
+  SCRIPT="quickstart/$SCRIPT"
+fi
+echo "Run $SCRIPT"
+
+docker run --rm \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env WEIGHT_PATH=${WEIGHT_PATH} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${WEIGHT_PATH}:${WEIGHT_PATH} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run_memrec.sh b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run_memrec.sh
new file mode 100644
index 000000000..ce0f78c3a
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/run_memrec.sh
@@ -0,0 +1,13 @@
+export MODEL_DIR="frameworks.ai.models.intel-models"
+
+# Env vars
+export PRECISION="fp32"
+export WEIGHT_PATH="memrec_CriteoTB_D75000dw75000_K1kw1.pt" #only needed for testing accuracy
+export DATASET_DIR="data/TB/preprocessed"
+export OUTPUT_DIR="memrec_out"
+
+# Run a quickstart script (for example, bare metal performance)
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu
+
+#bash inference_performance.sh
+bash accuracy.sh
diff --git a/quickstart/recommendation/pytorch/memrec_dlrm/requirements.txt b/quickstart/recommendation/pytorch/memrec_dlrm/requirements.txt
new file mode 100644
index 000000000..d0d50e986
--- /dev/null
+++ b/quickstart/recommendation/pytorch/memrec_dlrm/requirements.txt
@@ -0,0 +1,10 @@
+future==0.18.3
+numpy==1.26.0
+onnx==1.14.1
+pydot==1.4.2
+torch==2.0.1
+torchviz==0.0.2
+scikit-learn==1.3.0
+tqdm==4.66.1
+tensorboard==2.14.0
+wheel==0.38.1
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md
new file mode 100644
index 000000000..7589059ea
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md
@@ -0,0 +1,101 @@
+<!--- 0. Title -->
+# PyTorch DLRM inference
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running torchrec DLRM inference using
+Intel-optimized PyTorch for bare metal.
+
+## Bare Metal
+### General Setup
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, and Jemalloc.
+
+### Model Specific Setup
+
+* Install dependencies
+  ```bash
+  cd <clone of the model zoo>/quickstart/recommendation/pytorch/torchrec_dlrm
+  pip install requirements.txt
+  ```
+
+* Set Jemalloc Preload for better performance
+
+  The jemalloc should be built from the [General setup](#general-setup) section.
+  ```bash
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto"
+  ```
+
+* Set IOMP preload for better performance
+
+  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+  ```bash
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+  ```
+
+* Set ENV to use AMX if you are using SPR
+  ```bash
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+## Datasets
+The dataset can be downloaded and preprocessed by following https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset.
+We also provided a preprocessed scripts based on the instruction above. `preprocess_raw_dataset.sh`.
+After you loading the raw dataset `day_*.gz` and unzip them to RAW_DIR.
+```bash
+export MODEL_DIR=<where you clone this repo>
+export RAW_DIR=<the unziped raw dataset>
+export TEMP_DIR=<where your choose the put the temp file during preprocess>
+export PREPROCESSED_DIR=<where your choose the put the one-hot dataset>
+export MULTI_HOT_DIR=<where your choose the put the multi-hot dataset>
+bash preprocess_raw_dataset.sh
+```
+
+## Pre-Trained checkpoint
+Your can download and unzip checkpoint by following
+https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v2/pytorch#downloading-model-weights
+
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32, bf32, bf16, fp16, or int8). |
+| `test_accuracy.sh` | Run inference to verify auroc for the specified precision (fp32, bf32, bf16, fp16, or int8). |
+
+## Run the model
+
+```bash
+# Clone the model zoo repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+export OUTPUT_DIR=<specify the log dir to save log>
+
+# Env vars
+export PRECISION=<specify the precision to run>
+
+# Run a quickstart script for bare metal performance)
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu
+bash inference_performance.sh
+
+# Run a quickstart script for accuracy test
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu
+export DATASET_DIR=<multi-hot dataset dir>
+export WEIGHT_DIR=<offical released checkpoint>
+bash test_accuracy.sh
+```
+
+## Other Debug Env
+Setting following env can get memory-usage logs and pictures with `inference_performnace.sh` (Requires `memory_profiler` and `matplotlib`)
+```
+export PLOTMEM=true
+export MEMLOG=<mem-log-dir>
+export MEMPIC=<mem-picture-dir>
+``````
+
+<!--- 80. License -->
+## License
+
+[LICENSE](/LICENSE)
diff --git a/docker/pyt-cpu/Makefile b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_acc_all.sh
similarity index 56%
rename from docker/pyt-cpu/Makefile
rename to quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_acc_all.sh
index f0ae20dfb..ac3d40716 100644
--- a/docker/pyt-cpu/Makefile
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_acc_all.sh
@@ -1,6 +1,3 @@
-#
-# -*- coding: utf-8 -*-
-#
 # Copyright (c) 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -16,11 +13,20 @@
 # limitations under the License.
 #
 
-#
+export MODEL_DIR=/home/haozhe/lz/frameworks.ai.models.intel-models
+export WEIGHT_DIR=/data/dlrm2_weight/model_weights
+export DATASET_DIR=/data/dlrm_2_dataset/inference
+export OUTPUT_DIR=./
+export PLOTMEM=true
 
-PYT_BASE_IMAGE=intel/intel-extension-for-pytorch
-PYT_BASE_TAG=2.0.0-pip-base
-all:
-	@PYT_BASE_IMAGE=${PYT_BASE_IMAGE} \
-	 PYT_BASE_TAG=${PYT_BASE_TAG} \
-	 docker compose -f pyt-cpu-docker-compose.yml up --build
+for precision in int8 fp32 fp16 bf32 bf16
+do
+export PRECISION=$precision
+if [[ $PLOTMEM == "true" ]]; then
+export MEMLOG=./log/${PRECISION}-acc-mem.dat
+export MEMPIC=./log/${PRECISION}-acc-mem.jpeg
+rm $MEMLOG
+rm $MEMPIC
+fi
+bash test_accuracy.sh
+done
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_calibration.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_calibration.sh
new file mode 100644
index 000000000..564596dd3
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_calibration.sh
@@ -0,0 +1,51 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"  ]; then
+    echo "Could not find the script of dlrm_s_pytorch.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the dlrm_s_pytorch.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"
+    exit 1
+fi
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
+INT8_CONFIG=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+LOG_0="${LOG}/throughput.log"
+export BATCH_SIZE=32768
+python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc $MODEL_SCRIPT \
+    --embedding_dim 128 \
+    --dense_arch_layer_sizes 512,256,128 \
+    --over_arch_layer_sizes 1024,1024,512,256,1 \
+    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+    --epochs 1 \
+    --pin_memory \
+    --mmap_mode \
+    --batch_size $BATCH_SIZE \
+    --interaction_type=dcn \
+    --dcn_num_layers=3 \
+    --dcn_low_rank_dim=512 \
+    --ipex-optimize \
+    --inference-only \
+    --dtype int8 \
+    --int8-configure-dir ${INT8_CONFIG}\
+    --calibration \
+    --synthetic_multi_hot_criteo_path $DATASET_DIR \
+    --snapshot-dir $WEIGHT_DIR \
+    --ipex-merged-emb-cat
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_profile_all.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_profile_all.sh
new file mode 100644
index 000000000..171d9f13c
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/_profile_all.sh
@@ -0,0 +1,44 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export MODEL_DIR=/home/haozhe/lz/frameworks.ai.models.intel-models
+export DATASET_DIR=/data/dlrm_2_dataset/inference
+export OUTPUT_DIR=./
+export ENABLE_TORCH_PROFILE=true
+export PLOTMEM=true
+
+seq=`date +%m%d%H%M%S`
+mkdir log/${seq}/
+for precision in int8 fp32 fp16 bf32 bf16
+do
+export PRECISION=$precision
+if [[ $PLOTMEM == "true" ]]; then
+export MEMLOG=./log/${seq}/${PRECISION}-bench-real-mem.dat
+export MEMPIC=./log/${seq}/${PRECISION}-bench-real.jpeg
+fi
+bash inference_performance.sh
+done
+
+
+unset DATASET_DIR
+for precision in int8 fp32 fp16 bf32 bf16
+do
+export PRECISION=$precision
+if [[ $PLOTMEM == "true" ]]; then
+export MEMLOG=./log/${seq}/${PRECISION}-bench-dummy-mem.dat
+export MEMPIC=./log/${seq}/${PRECISION}-bench-dummy-mem.jpeg
+fi
+bash inference_performance.sh 2>&1 |tee ./log/${PRECISION}-bench-dummy.log
+done
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh
new file mode 100644
index 000000000..7ec3df9f0
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh
@@ -0,0 +1,133 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"  ]; then
+    echo "Could not find the script of dlrm_s_pytorch.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the dlrm_s_pytorch.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"
+    exit 1
+fi
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
+INT8_CONFIG=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
+
+echo "PRECISION: ${PRECISION}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+LOG=${OUTPUT_DIR}/dlrm_inference_performance_log/${PRECISION}
+rm -rf ${LOG}
+mkdir -p ${LOG}
+
+ARGS=""
+if [[ $PRECISION == "bf16" ]]; then
+    ARGS="$ARGS --dtype bf16 --ipex-merged-emb-cat"
+    echo "running bf16 path"
+elif [[ $PRECISION == "fp32" ]]; then
+    echo "running fp32 path"
+    ARGS="$ARGS --dtype fp32 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "bf32" ]]; then
+    echo "running bf32 path"
+    ARGS="$ARGS --dtype bf32 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "fp16" ]]; then
+    echo "running fp16 path"
+    ARGS="$ARGS --dtype fp16 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "int8" ]]; then
+    echo "running int8 path"
+    ARGS="$ARGS --dtype int8 --ipex-merged-emb-cat --int8-configure-dir ${INT8_CONFIG}"
+else
+    echo "The specified PRECISION '${PRECISION}' is unsupported."
+    echo "Supported PRECISIONs are: fp32, fp16, bf16, bf32, int8"
+    exit 1
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "DATASET_DIR are not set, will use dummy generated dataset"
+  ARGS="$ARGS --multi_hot_distribution_type uniform "
+  ARGS="$ARGS --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 "
+else
+  ARGS="$ARGS --synthetic_multi_hot_criteo_path $DATASET_DIR "
+fi
+
+LOG_0="${LOG}/throughput.log"
+
+if [ -z "${BATCH_SIZE}" ]; then
+  BATCH_SIZE=512
+fi
+
+export launcher_arg="-m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc"
+if [[ $PLOTMEM == "true" ]]; then
+pip install memory_profiler matplotlib
+export mrun_cmd="mprof run --python -o ${MEMLOG}"
+unset launcher_arg
+fi
+
+if [[ $ENABLE_TORCH_PROFILE == "true" ]]; then
+  ARGS="$ARGS --profile"
+fi
+
+CORES_PER_SOCKET=`lscpu | grep "Core(s) per socket" | awk '{print $4}'`
+SOCKETS=`lscpu | grep "Socket(s)" | awk '{print $2}'`
+NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
+NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
+CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
+
+export OMP_NUM_THREADS=1
+
+$mrun_cmd python $launcher_arg $MODEL_SCRIPT \
+    --embedding_dim 128 \
+    --dense_arch_layer_sizes 512,256,128 \
+    --over_arch_layer_sizes 1024,1024,512,256,1 \
+    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+    --epochs 1 \
+    --pin_memory \
+    --mmap_mode \
+    --batch_size $BATCH_SIZE \
+    --interaction_type=dcn \
+    --dcn_num_layers=3 \
+    --dcn_low_rank_dim=512 \
+    --limit_val_batches 1000 \
+    --ipex-optimize \
+    --log-freq 10 \
+    --jit \
+    --inference-only \
+    --benchmark \
+    --share-weight-instance=$CORES_PER_NUMA_NODE \
+    $ARGS 2>&1 | tee $LOG_0
+wait
+
+if [[ $PLOTMEM == "true" ]]; then
+mprof plot ${MEMLOG} -o ${MEMPIC}
+fi
+
+throughput=$(grep 'Throughput:' ${LOG}/throughput.log |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
+BEGIN {
+        sum = 0;
+        i = 0;
+      }
+      {
+        sum = sum + $1;
+        i++;
+      }
+END   {
+sum = sum / i;
+        printf("%.3f", sum);
+}')
+echo ""dlrm-v2";"throughput";${PRECISION};${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/preprocess_raw_dataset.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/preprocess_raw_dataset.sh
new file mode 100644
index 000000000..823d9c7e4
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/preprocess_raw_dataset.sh
@@ -0,0 +1,44 @@
+MODEL_DIR=${MODEL_DIR-$PWD}
+PREPRECESS_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
+GET_MULTI_HOT_SCRIPTS=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py
+
+if [ ! -e "$PREPRECESS_SCRIPT"  ]; then
+    echo "Could not find the script of process_Criteo_1TB_Click_Logs_dataset.sh. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the process_Criteo_1TB_Click_Logs_dataset.sh exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh"
+    exit 1
+fi
+
+if [ ! -e "$GET_MULTI_HOT_SCRIPTS"  ]; then
+    echo "Could not find the script of materialize_synthetic_multihot_dataset.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the materialize_synthetic_multihot_dataset.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.sh"
+    exit 1
+fi
+
+if [ -z "${RAW_DIR}" ]; then
+  echo "The required environment variable RAW_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${TEMP_DIR}" ]; then
+  echo "The required environment variable TEMP_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${PREPROCESSED_DIR}" ]; then
+  echo "The required environment variable PREPROCESSED_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${MULTI_HOT_DIR}" ]; then
+  echo "The required environment variable MULTI_HOT_DIR has not been set"
+  exit 1
+fi
+
+bash $PREPRECESS_SCRIPT ${RAW_DIR} ${TEMP_DIR} ${PREPROCESSED_DIR}
+
+python $GET_MULTI_HOT_SCRIPTS \
+    --in_memory_binary_criteo_path $PREPROCESSED_DIR \
+    --output_path $MULTI_HOT_DIR \
+    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+    --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+    --multi_hot_distribution_type uniform
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh
new file mode 100644
index 000000000..7d8f68275
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh
@@ -0,0 +1,104 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"  ]; then
+    echo "Could not find the script of dlrm_s_pytorch.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the dlrm_s_pytorch.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"
+    exit 1
+fi
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
+INT8_CONFIG=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
+
+echo "PRECISION: ${PRECISION}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+fi
+
+if [ -z "${WEIGHT_DIR}" ]; then
+  echo "The required environment variable WEIGHT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+LOG=${OUTPUT_DIR}/dlrm_inference_accuarcy_log/${PRECISION}
+rm -rf ${LOG}
+mkdir -p ${LOG}
+
+ARGS=""
+if [[ $PRECISION == "bf16" ]]; then
+    ARGS="$ARGS --dtype bf16 --ipex-merged-emb-cat"
+    echo "running bf16 path"
+elif [[ $PRECISION == "fp32" ]]; then
+    echo "running fp32 path"
+    ARGS="$ARGS --dtype fp32 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "bf32" ]]; then
+    echo "running bf32 path"
+    ARGS="$ARGS --dtype bf32 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "fp16" ]]; then
+    echo "running fp16 path"
+    ARGS="$ARGS --dtype fp16 --ipex-merged-emb-cat"
+elif [[ $PRECISION == "int8" ]]; then
+    echo "running int8 path"
+    ARGS="$ARGS --dtype int8 --ipex-merged-emb-cat --int8-configure-dir ${INT8_CONFIG}"
+else
+    echo "The specified PRECISION '${PRECISION}' is unsupported."
+    echo "Supported PRECISIONs are: fp32, bf32, fp16, bf16, int8"
+    exit 1
+fi
+
+LOG_0="${LOG}/acc.log"
+export BATCH_SIZE=65536
+
+if [[ $PLOTMEM == "true" ]]; then
+pip install memory_profiler
+export mrun_cmd="mprof run --python -o ${MEMLOG}"
+fi
+
+# Do not need to use launcher to bind memory for accuracy test
+$mrun_cmd python $MODEL_SCRIPT \
+    --snapshot-dir $WEIGHT_DIR \
+    --embedding_dim 128 \
+    --dense_arch_layer_sizes 512,256,128 \
+    --over_arch_layer_sizes 1024,1024,512,256,1 \
+    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+    --epochs 1 \
+    --pin_memory \
+    --mmap_mode \
+    --batch_size $BATCH_SIZE \
+    --interaction_type=dcn \
+    --dcn_num_layers=3 \
+    --dcn_low_rank_dim=512 \
+    --ipex-optimize \
+    --inference-only \
+    --synthetic_multi_hot_criteo_path $DATASET_DIR \
+    --test_auroc \
+    $ARGS 2>&1 | tee $LOG_0
+
+if [[ $PLOTMEM == "true" ]]; then
+mprof plot ${MEMLOG} -o ${MEMPIC}
+fi
+
+accuracy=$(grep 'Final AUROC:' $LOG_0 |sed -e 's/.*Final AUROC//;s/[^0-9.]//g')
+echo ""dlrm-v2";"auc";${PRECISION};${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md
new file mode 100644
index 000000000..61f7e485c
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md
@@ -0,0 +1,68 @@
+# PyTorch DLRM Inference
+
+## Description 
+This document has instructions for running DLRM Inference with FP16 precision on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch.
+
+## Datasets
+
+The dataset required for inference is the [Criteo 1TB Click Logs dataset](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf). Please refer to the [link](https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset) to follow steps 1-3. After dataset pre-processing, set the `DATASET_DIR` environment variable to point to the dataset directory. Please note that the pre-processing step requires 700GB of RAM and takes 1-2 days to run.
+
+## Pre-trained Model
+
+Download the pre-trained model as follows:
+```bash
+wget https://cloud.mlcommons.org/index.php/s/XzfSeLgW8FYfR3S/download -O weigths.zip
+unzip weights.zip
+```
+After decompressing the zip file, set `PRETRAINED_MODEL` to the pre-trained model directory. 
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+|`multi_card_distributed_inference .sh` | Runs DLRM distributed inference on single-node x4 OAM Modules |
+
+Requirements:
+* Host machine has Intel(R) Data Center Max Series 1550 x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+* Docker
+
+## Docker pull Command
+
+```bash
+docker pull intel/recommendation:pytorch-max-gpu-dlrm-inference
+```
+The DLRM inference container includes scripts,models,libraries needed to run fp16 inference. To run the `multi_card_distributed_inference.sh` quickstart script follow the instructions below. 
+```bash
+export GLOBAL_BATCH_SIZE=<provide suitable batch size. Default is 65536>
+export TOTAL_SAMPLES=<provide suitable number. Default is 4195197692>
+export PRECISION=<provide suitable precision. Default is fp16>
+export PRETRAINED_MODEL=<provide path to the pre-trained model directory>
+export DATASET_DIR=<provide path to the Terabyte Dataset directory>
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export OUTPUT_DIR=<path to output directory to view logs> 
+ 
+DOCKER_ARGS="--rm -it"
+IMAGE_NAME=intel/recommendation:pytorch-max-gpu-dlrm-inference
+SCRIPT=quickstart/multi_card_distributed_inference.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env NUM_OAM=${NUM_OAM} \
+  --env PRECISION=${PRECISION} \
+  --env PRETRAINED_MODEL=${PRETRAINED_MODEL} \
+  --env GLOBAL_BATCH_SIZE=${GLOBAL_BATCH_SIZE} \
+  --env TOTAL_SAMPLES=${TOTAL_SAMPLES} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume /dev/dri:/dev/dri \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${PRETRAINED_MODEL}:${PRETRAINED_MODEL} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/README.md
new file mode 100644
index 000000000..12931d01c
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/README.md
@@ -0,0 +1,92 @@
+<!--- 0. Title -->
+# PyTorch DLRM Inference for Intel(R) Data Center GPU Max Series
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running DLRM Inference with FP16 precision on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch.
+
+<!--- 20. GPU Setup -->
+## Software Requirements:
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-pytorch/) to install the latest IPEX version and other prerequisites.
+
+- Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
+  - Intel® oneAPI DPC++ Compiler
+  - Intel® oneAPI Threading Building Blocks (oneTBB)
+  - Intel® oneAPI Math Kernel Library (oneMKL)
+  - Follow [instructions](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=offline) to download and install the latest oneAPI Base Toolkit.
+
+  - Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+
+<!--- 30. Datasets -->
+## Datasets
+
+The dataset required for inference is the [Criteo 1TB Click Logs dataset](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf). Please refer to the [link](https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset) to follow steps 1-3. After dataset pre-processing, set the `DATASET_DIR` environment variable to point to the dataset directory. Please note that the pre-processing step requires 700GB of RAM and takes 1-2 days to run.
+
+## Pre-trained Model
+
+Download the pre-trained model as follows:
+```bash
+wget https://cloud.mlcommons.org/index.php/s/XzfSeLgW8FYfR3S/download -O weigths.zip
+unzip weights.zip
+```
+After decompressing the zip file, set `PRETRAINED_MODEL` to the pre-trained model directory. 
+
+<!--- 40. Quick Start Scripts -->
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+|`multi_card_distributed_inference .sh` | Runs DLRM distributed inference on single-node x4 OAM Modules |
+
+<!--- 50. Baremetal -->
+## Run the model
+* Clone the AI Reference Models repository:
+  ```bash
+  git clone https://github.com/IntelAI/models.git
+  ```
+* Navigate to models directory and install model specific dependencies for the workload:
+  ```bash
+  cd models
+  # Install model specific dependencies:
+  ./quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/setup.sh
+  ```
+
+See the [datasets section](#datasets) of this document for instructions on
+downloading the dataset.
+
+```
+# Set environment vars:
+export PRETRAINED_MODEL=<provide path to the pre-trained model directory>
+export DATASET_DIR=<provide path to the Terabyte Dataset directory>
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export OUTPUT_DIR=<path to output directory to view logs> 
+export PRECISION=<provide suitable precision, currently supports fp16 precision> 
+
+# Optional envs
+export GLOBAL_BATCH_SIZE=<provide suitable batch size. Default is 65536>
+export TOTAL_SAMPLES=<provide suitable number. Default is 4195197692>
+
+# Run a quickstart script
+./quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_card_distributed_inference.sh
+```
+
+<!--- 80. License -->
+## License
+
+[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_card_distributed_inference.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_card_distributed_inference.sh
new file mode 100755
index 000000000..67f8c1813
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/multi_card_distributed_inference.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+GLOBAL_BATCH_SIZE=${GLOBAL_BATCH_SIZE-65536}
+TOTAL_SAMPLES=${TOTAL_SAMPLES-4195197692}
+NUM_NODES=${NUM_NODES-1}
+SHARDING_PLAN=${SHARDING_PLAN-round_robin}
+
+if [[ -z "$PRETRAINED_MODEL" ]]; then
+    echo "The required environment variable $PRETRAINED_MODEL has not been set"
+    exit 1
+fi 
+
+if [[ -z $PRECISION ]]; then
+  echo "The required environment variable PRECISION has not been set"
+  exit 1
+fi
+
+if [[ -z $OUTPUT_DIR ]]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+if [[ -z "${DATASET_DIR}" ]]; then
+    echo "The required environment variable DATASET_DIR has not been set"
+    exit 1
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "${DATASET_DIR} does not exist."
+    exit 1
+fi
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+if [[ -z $NUM_OAM ]]; then
+  echo "The required environment variable NUM_OAM has not been set."
+  exit 1
+fi
+
+source /opt/intel/oneapi/setvars.sh
+export ONECCL_BINDINGS_FOR_PYTORCH_ENV_VERBOSE=1
+export MASTER_ADDR='127.0.0.1'
+
+echo "Running DLRM single-node multi-card inference on ${NUM_OAM} OAM Modules"
+
+ARGS+=" --embedding_dim 128"
+ARGS+=" --dense_arch_layer_sizes 512,256,128"
+ARGS+=" --over_arch_layer_sizes 1024,1024,512,256,1"
+ARGS+=" --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36"
+ARGS+=" --validation_freq_within_epoch $((TOTAL_SAMPLES / (GLOBAL_BATCH_SIZE * 20 * 1000)))"
+ARGS+=" --synthetic_multi_hot_criteo_path $DATASET_DIR"
+ARGS+=" --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1"
+#ARGS+=" --multi_hot_distribution_type uniform"
+ARGS+=" --use_xpu"
+ARGS+=" --epochs 1"
+ARGS+=" --pin_memory"
+ARGS+=" --mmap_mode"
+ARGS+=" --batch_size $GLOBAL_BATCH_SIZE"
+ARGS+=" --interaction_type=dcn"
+ARGS+=" --dcn_num_layers=3"
+ARGS+=" --adagrad"
+ARGS+=" --dcn_low_rank_dim=512"
+ARGS+=" --numpy_rand_seed=12345"
+ARGS+=" --log_freq 10"
+ARGS+=" --amp"
+ARGS+=" --inference_only"
+ARGS+=" --snapshot_dir ${PRETRAINED_MODEL}"
+ARGS+=" --num_batches 50"
+ARGS+=" --sharding_plan ${SHARDING_PLAN}"
+ARGS+=" --num_nodes ${NUM_NODES}"
+ARGS+=" --learning_rate 0.005"
+
+if [[ "$PRECISION" == "fp16" ]];then
+    ARGS+=" --fp16"
+    echo $ARGS 
+else
+    echo "DLRM Inference workload currently supports fp16 precision"
+    exit 1
+fi
+
+if [[ "${NUM_OAM}" == "4" ]]; then
+    np=8
+    ppn=$np
+    ZE_AFFINITY_MASK="0.0,0.1,1.0,1.1,2.0,2.1,3.0,3.1"
+else
+    echo "Currently only x4 OAM Modules are supported."
+    exit 1
+fi 
+
+cd ${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/inference/gpu
+I_MPI_DEBUG=6 ZE_AFFINITY_MASK=$ZE_AFFINITY_MASK mpirun -np $np -ppn $ppn --prepend-rank python -u dlrm_main.py ${ARGS} 2>&1 | tee ${OUTPUT_DIR}/dlrm_x${NUM_OAM}_inference.log
diff --git a/docker/max-gpu/Makefile b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/setup.sh
old mode 100644
new mode 100755
similarity index 56%
rename from docker/max-gpu/Makefile
rename to quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/setup.sh
index b152ff388..067206385
--- a/docker/max-gpu/Makefile
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/setup.sh
@@ -1,6 +1,3 @@
-#
-# -*- coding: utf-8 -*-
-#
 # Copyright (c) 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -14,19 +11,15 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+# ============================================================================
 #
-
+# THIS IS A GENERATED DOCKERFILE.
 #
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information
 
-MAX_TF_BASE_IMAGE=intel/intel-extension-for-tensorflow
-MAX_TF_BASE_TAG=gpu-horovod
-MAX_PYT_BASE_IMAGE=intel/intel-extension-for-pytorch
-MAX_PYT_BASE_TAG=xpu-max
-
-all:
-	@MAX_PYT_BASE_IMAGE=${MAX_PYT_BASE_IMAGE} \
-	 MAX_PYT_BASE_TAG=${MAX_PYT_BASE_TAG} \
-	 MAX_TF_BASE_IMAGE=${MAX_TF_BASE_IMAGE} \
-	 MAX_TF_BASE_TAG=${MAX_TF_BASE_TAG} \
-	 docker compose -f max-gpu-docker-compose.yml up --build
+apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing numactl
 
+pip install -r models/recommendation/pytorch/torchrec_dlrm/inference/gpu/requirements.txt
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/requirements.txt b/quickstart/recommendation/pytorch/torchrec_dlrm/requirements.txt
new file mode 100644
index 000000000..95a188cc2
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/requirements.txt
@@ -0,0 +1,3 @@
+torchmetrics==0.11.0
+torchrec==0.3.2
+torchsnapshot
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md
new file mode 100644
index 000000000..c5816c492
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md
@@ -0,0 +1,71 @@
+<!--- 0. Title -->
+# PyTorch DLRM training
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running torchrec DLRM training using
+Intel-optimized PyTorch for bare metal.
+
+## Bare Metal
+### General Setup
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, and Jemalloc.
+
+### Model Specific Setup
+
+* Install dependencies
+  ```bash
+  cd <clone of the model zoo>/quickstart/recommendation/pytorch/torchrec_dlrm
+  pip install requirements.txt
+  ```
+
+* Set Jemalloc Preload for better performance
+
+  The jemalloc should be built from the [General setup](#general-setup) section.
+  ```bash
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto"
+  ```
+
+* Set IOMP preload for better performance
+
+  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+  ```bash
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+  ```
+
+* Set ENV to use AMX if you are using SPR
+  ```bash
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+## Datasets
+
+Use random dataset.
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `training_performance.sh` | Run training to verify performance for the specified precision (fp32, bf32, fp16, or bf16). |
+
+## Run the model
+
+```bash
+# Clone the model zoo repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+export OUTPUT_DIR=<specify the log dir to save log>
+# Env vars
+export PRECISION=<specify the precision to run>
+
+# Run a quickstart script (for example, bare metal performance)
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu
+bash training_performance.sh
+```
+
+<!--- 80. License -->
+## License
+
+[LICENSE](/LICENSE)
diff --git a/docker/flex-gpu/Makefile b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/_profile_all.sh
similarity index 54%
rename from docker/flex-gpu/Makefile
rename to quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/_profile_all.sh
index 5c521c29b..6cb3ea38e 100644
--- a/docker/flex-gpu/Makefile
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/_profile_all.sh
@@ -1,6 +1,3 @@
-#
-# -*- coding: utf-8 -*-
-#
 # Copyright (c) 2023 Intel Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -16,17 +13,18 @@
 # limitations under the License.
 #
 
-#
-
-FLEX_TF_BASE_IMAGE=intel/intel-extension-for-tensorflow
-FLEX_TF_BASE_TAG=gpu
-FLEX_PYT_BASE_IMAGE=intel/intel-extension-for-pytorch
-FLEX_PYT_BASE_TAG=xpu-flex
-
-all:
-	@FLEX_PYT_BASE_IMAGE=${FLEX_PYT_BASE_IMAGE} \
-	 FLEX_PYT_BASE_TAG=${FLEX_PYT_BASE_TAG} \
-	 FLEX_TF_BASE_IMAGE=${FLEX_TF_BASE_IMAGE} \
-	 FLEX_TF_BASE_TAG=${FLEX_TF_BASE_TAG} \
-	 docker compose -f flex-gpu-docker-compose.yml up --build
-
+export MODEL_DIR=/home/haozhe/lz/frameworks.ai.models.intel-models
+export OUTPUT_DIR=./
+export ENABLE_TORCH_PROFILE=true
+export PLOTMEM=true
+seq=`date +%m%d%H%M%S`
+mkdir log/${seq}/
+for precision in fp32 bf32 fp16 bf16
+do
+export PRECISION=$precision
+if [[ $PLOTMEM == "true" ]]; then
+export MEMLOG=./log/${seq}/${PRECISION}-bench-dummy-mem.dat
+export MEMPIC=./log/${seq}/${PRECISION}-bench-dummy-mem.jpeg
+fi
+bash training_performance.sh 2>&1 |tee ./log/${seq}/${PRECISION}-bench-dummy.log
+done
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh
new file mode 100644
index 000000000..fee8de5a9
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh
@@ -0,0 +1,110 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+if [ ! -e "${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"  ]; then
+    echo "Could not find the script of dlrm_s_pytorch.py. Please set environment variable '\${MODEL_DIR}'."
+    echo "From which the dlrm_s_pytorch.py exist at the: \${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py"
+    exit 1
+fi
+MODEL_SCRIPT=${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
+
+echo "PRECISION: ${PRECISION}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+LOG=${OUTPUT_DIR}/dlrm_training_performance_log/${PRECISION}
+rm -rf ${LOG}
+mkdir -p ${LOG}
+
+ARGS=""
+if [[ $PRECISION == "bf16" ]]; then
+    ARGS="$ARGS --dtype bf16"
+    echo "running bf16 path"
+elif [[ $PRECISION == "fp32" ]]; then
+    echo "running fp32 path"
+    ARGS="$ARGS --dtype fp32"
+elif [[ $PRECISION == "bf32" ]]; then
+    echo "running bf32 path"
+    ARGS="$ARGS --dtype bf32"
+elif [[ $PRECISION == "fp16" ]]; then
+    echo "running fp16 path"
+    ARGS="$ARGS --dtype fp16"
+else
+    echo "The specified PRECISION '${PRECISION}' is unsupported."
+    echo "Supported PRECISIONs are: fp32, bf32, fp16, bf16"
+    exit 1
+fi
+
+if [[ $ENABLE_TORCH_PROFILE == "true" ]]; then
+  ARGS="$ARGS --profile"
+fi
+
+export launcher_arg="-m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc"
+if [[ $PLOTMEM == "true" ]]; then
+pip install memory_profiler matplotlib
+export mrun_cmd="mprof run --python -o ${MEMLOG}"
+unset launcher_arg
+fi
+
+LOG_0="${LOG}/throughput.log"
+export BATCH_SIZE=16
+$mrun_cmd python $launcher_arg $MODEL_SCRIPT \
+    --embedding_dim 128 \
+    --dense_arch_layer_sizes 512,256,128 \
+    --over_arch_layer_sizes 1024,1024,512,256,1 \
+    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+    --epochs 1 \
+    --pin_memory \
+    --mmap_mode \
+    --batch_size $BATCH_SIZE \
+    --interaction_type=dcn \
+    --dcn_num_layers=3 \
+    --dcn_low_rank_dim=512 \
+    --adagrad \
+    --learning_rate 0.005 \
+    --multi_hot_distribution_type uniform \
+    --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+    --limit_train_batches 100 \
+    --ipex-optimize \
+    --log-freq 10 \
+    --benchmark \
+    $ARGS 2>&1 | tee $LOG_0
+wait
+
+if [[ $PLOTMEM == "true" ]]; then
+mprof plot ${MEMLOG} -o ${MEMPIC}
+fi
+
+throughput=$(grep 'Throughput:' ${LOG}/throughput.log |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
+BEGIN {
+        sum = 0;
+        i = 0;
+      }
+      {
+        sum = sum + $1;
+        i++;
+      }
+END   {
+sum = sum / i;
+        printf("%.3f", sum);
+}')
+echo ""dlrm-v2";"training throughput";${PRECISION};${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md
new file mode 100644
index 000000000..f598b151b
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md
@@ -0,0 +1,57 @@
+# PyTorch DLRM Training
+
+## Description 
+This document has instructions for running DLRM training with BFloat16 precision on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch.
+
+## Datasets
+
+The dataset required to train the model is the [Criteo 1TB Click Logs dataset](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf). Please refer to the [link](https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset) to follow steps 1-3. After dataset pre-processing, set the `DATASET_DIR` environment variable to point to the dataset directory. Please note that the pre-processing step requires 700GB of RAM and takes 1-2 days to run.
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+|`multi_card_distributed_train.sh` | Runs DLRM distributed training on single-node x4 OAM Modules |
+
+Requirements:
+* Host machine has Intel(R) Data Center Max Series 1550 x4 OAM GPU
+* Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+* Docker
+
+## Docker pull Command
+
+```bash
+docker pull intel/recommendation:pytorch-max-gpu-dlrm-training
+```
+The DLRM training container includes scripts,models,libraries needed to run bf16 training. To run the `multi_card_distributed_train.sh` quickstart script follow the instructions below. 
+
+```bash
+export GLOBAL_BATCH_SIZE=<provide suitable batch size. Default is 65536>
+export TOTAL_TRAINING_SAMPLES=<provide suitable number. Default is 4195197692>
+export PRECISION=<provide suitable precision. Default is bf16>
+export DATASET_DIR=<provide path to the Terabyte Dataset directory>
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export OUTPUT_DIR=<path to output directory to view logs> 
+
+DOCKER_ARGS="--rm -it"
+IMAGE_NAME=intel/recommendation:pytorch-max-gpu-dlrm-training
+SCRIPT=quickstart/multi_card_distributed_train.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env NUM_OAM=${NUM_OAM} \
+  --env PRECISION=${PRECISION} \
+  --env GLOBAL_BATCH_SIZE=${GLOBAL_BATCH_SIZE} \
+  --env TOTAL_TRAINING_SAMPLES=${TOTAL_TRAINING_SAMPLES} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume /dev/dri:/dev/dri \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/README.md
new file mode 100644
index 000000000..29792c650
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/README.md
@@ -0,0 +1,82 @@
+<!--- 0. Title -->
+# PyTorch DLRM Training for Intel(R) Data Center GPU Max Series
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running DLRM training with BFloat16 precision on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch.
+
+<!--- 20. GPU Setup -->
+## Software Requirements:
+- Host machine has Intel® Data Center Max Series 1550 x4 OAM GPU
+- Follow instructions to install GPU-compatible driver [647](https://dgpu-docs.intel.com/releases/stable_647_21_20230714.html)
+- Create and activate virtual environment.
+  ```bash
+  virtualenv -p python <virtualenv_name>
+  source <virtualenv_name>/bin/activate
+  ```
+- Follow [instructions](https://pypi.org/project/intel-extension-for-pytorch/) to install the latest IPEX version and other prerequisites.
+
+- Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
+  - Intel® oneAPI DPC++ Compiler
+  - Intel® oneAPI Threading Building Blocks (oneTBB)
+  - Intel® oneAPI Math Kernel Library (oneMKL)
+  - Follow [instructions](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=offline) to download and install the latest oneAPI Base Toolkit.
+
+  - Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+
+<!--- 30. Datasets -->
+## Datasets
+
+The dataset required to train the model is the [Criteo 1TB Click Logs dataset](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf). Please refer to the [link](https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset) to follow steps 1-3. After dataset pre-processing, set the `DATASET_DIR` environment variable to point to the dataset directory. Please note that the pre-processing step requires 700GB of RAM and takes 1-2 days to run.
+
+<!--- 40. Quick Start Scripts -->
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+|`multi_card_distributed_train.sh` | Runs DLRM distributed training on single-node x4 OAM Modules |
+
+<!--- 50. Baremetal -->
+## Run the model
+* Clone the AI Reference Models repository:
+  ```bash
+  git clone https://github.com/IntelAI/models.git
+  ```
+* Navigate models directory and install model specific dependencies for the workload:
+  ```bash
+  cd models
+  # Install model specific dependencies:
+  ./quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/setup.sh
+  ```
+
+See the [datasets section](#datasets) of this document for instructions on
+downloading the dataset.
+
+```
+# Set environment vars:
+export DATASET_DIR=<provide path to the Terabyte Dataset directory>
+export NUM_OAM=<provide 4 for number of OAM Modules supported by the platform>
+export OUTPUT_DIR=<path to output directory to view logs>
+export PRECISION=<provide suitable precision, currently supports bf16 precision> 
+
+# Optional envs
+export GLOBAL_BATCH_SIZE=<provide suitable batch size. Default is 65536>
+export TOTAL_TRAINING_SAMPLES=<provide suitable number. Default is 4195197692>
+
+# Run a quickstart script
+./quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_card_distributed_train.sh
+```
+
+<!--- 80. License -->
+## License
+
+[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_card_distributed_train.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_card_distributed_train.sh
new file mode 100755
index 000000000..80000bab5
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/multi_card_distributed_train.sh
@@ -0,0 +1,100 @@
+#!/bin/bash
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+GLOBAL_BATCH_SIZE=${GLOBAL_BATCH_SIZE-65536}
+TOTAL_TRAINING_SAMPLES=${TOTAL_TRAINING_SAMPLES-4195197692}
+NUM_NODES=${NUM_NODES-1}
+SHARDING_PLAN=${SHARDING_PLAN-round_robin}
+
+if [[ -z $PRECISION ]]; then
+  echo "The required environment variable PRECISION has not been set"
+  exit 1
+fi
+
+if [[ -z $OUTPUT_DIR ]]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+if [[ -z $NUM_OAM ]]; then
+  echo "The required environment variable NUM_OAM has not been set."
+  exit 1
+fi
+
+if [[ -z "${DATASET_DIR}" ]]; then
+    echo "The required environment variable DATASET_DIR has not been set"
+    exit 1
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "${DATASET_DIR} does not exist."
+    exit 1
+fi
+
+source /opt/intel/oneapi/setvars.sh
+export ONECCL_BINDINGS_FOR_PYTORCH_ENV_VERBOSE=1
+export MASTER_ADDR='127.0.0.1'
+
+echo "Running DLRM single-node distributed multi-card training on ${NUM_OAM} OAM Modules"
+
+ARGS+=" --embedding_dim 128"
+ARGS+=" --dense_arch_layer_sizes 512,256,128"
+ARGS+=" --over_arch_layer_sizes 1024,1024,512,256,1"
+ARGS+=" --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36"
+ARGS+=" --validation_freq_within_epoch $((TOTAL_TRAINING_SAMPLES / (GLOBAL_BATCH_SIZE * 20 * 1000)))"
+ARGS+=" --synthetic_multi_hot_criteo_path $DATASET_DIR"
+ARGS+=" --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1"
+#ARGS+=" --multi_hot_distribution_type uniform"
+ARGS+=" --use_xpu"
+ARGS+=" --epochs 1"
+ARGS+=" --pin_memory"
+ARGS+=" --mmap_mode"
+ARGS+=" --batch_size $GLOBAL_BATCH_SIZE"
+ARGS+=" --interaction_type=dcn"
+ARGS+=" --dcn_num_layers=3"
+ARGS+=" --adagrad"
+ARGS+=" --dcn_low_rank_dim=512"
+ARGS+=" --numpy_rand_seed=12345"
+ARGS+=" --log_freq 10"
+ARGS+=" --amp"
+ARGS+=" --num_batches 50"
+ARGS+=" --sharding_plan ${SHARDING_PLAN}"
+ARGS+=" --num_nodes ${NUM_NODES}"
+ARGS+=" --learning_rate 0.005"
+
+if [[ "${NUM_OAM}" == "4" ]]; then
+    np=8
+    ppn=$np
+    ZE_AFFINITY_MASK="0.0,0.1,1.0,1.1,2.0,2.1,3.0,3.1"
+else
+    echo "Currently only x4 OAM Modules are supported"
+    exit 1
+fi 
+
+if [[ ${PRECISION} == "bf16" ]]; then 
+    ARGS+=" --bf16"
+    echo "${ARGS}"
+else
+    echo "DLRM Training currently supports bf16 precision"
+    exit 1
+fi
+
+cd ${MODEL_DIR}/models/recommendation/pytorch/torchrec_dlrm/training/gpu
+I_MPI_DEBUG=6 ZE_AFFINITY_MASK=$ZE_AFFINITY_MASK mpirun -np $np -ppn $ppn --prepend-rank python -u dlrm_main.py ${ARGS} 2>&1 | tee ${OUTPUT_DIR}/dlrm_x${NUM_OAM}_training.log
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/setup.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/setup.sh
new file mode 100755
index 000000000..673b0a835
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/setup.sh
@@ -0,0 +1,26 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information
+
+apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing numactl
+
+pip install -r models/recommendation/pytorch/torchrec_dlrm/training/gpu/requirements.txt
+pip install -e git+https://github.com/mlperf/logging#egg=mlperf-logging
diff --git a/quickstart/recommendation/tensorflow/dien/training/cpu/README_SPR_Container.md b/quickstart/recommendation/tensorflow/dien/training/cpu/README_SPR_Container.md
deleted file mode 100644
index d4074fa28..000000000
--- a/quickstart/recommendation/tensorflow/dien/training/cpu/README_SPR_Container.md
+++ /dev/null
@@ -1,65 +0,0 @@
-<!--- 0. Title -->
-# DIEN Training
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running DIEN training using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Use [prepare_data.sh](https://github.com/alibaba/ai-matrix/blob/master/macro_benchmark/DIEN_TF2/prepare_data.sh) to get [a subset of the Amazon book reviews data](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/) and process it.
-Or download and extract the preprocessed data files directly:
-```
-wget https://zenodo.org/record/3463683/files/data.tar.gz
-wget https://zenodo.org/record/3463683/files/data1.tar.gz
-wget https://zenodo.org/record/3463683/files/data2.tar.gz
-
-tar -jxvf data.tar.gz
-mv data/* .
-tar -jxvf data1.tar.gz
-mv data1/* .
-tar -jxvf data2.tar.gz
-mv data2/* .
-```
-Set the `DATASET_DIR` to point to the directory with the dataset files when running DIEN.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`training.sh`](/quickstart/recommendation/tensorflow/dien/training/cpu/training.sh) | Runs training with a batch size of 128 for the specified precision fp32. |
-
-## Run the model
-
-After you've followed the instructions to [build the container](#build-the-container)
-and [prepare the dataset](#datasets), use the `run.sh` script from the container
-package to run DIEN training in docker. Set environment variables to
-specify the dataset directory, precision to run, and
-an output directory. 
-By default, the `run.sh` script will run the
-`training.sh` quickstart script. To run a different script, specify
-the name of the script using the `SCRIPT` environment variable.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export PRECISION=<specify the precision to run>
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run the container with training.sh quickstart script
-./run.sh
-```
-
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
-
diff --git a/tests/cicd/PyTorch/resnet50-inference/run_test.sh b/tests/cicd/PyTorch/resnet50-inference/run_test.sh
index 3321cdc46..1bba7a070 100755
--- a/tests/cicd/PyTorch/resnet50-inference/run_test.sh
+++ b/tests/cicd/PyTorch/resnet50-inference/run_test.sh
@@ -5,23 +5,24 @@ echo "Setup PyTorch Test Enviroment for ResNet50 Inference"
 
 PRECISION=$1
 SCRIPT=$2
-DATASET=$3
-OUTPUT_DIR=${OUTPUT_DIR-"tests/cicd/PyTorch/output/resnet50-inference/${PRECISION}"}
-WORKSPACE=$4
-is_lkg_drop=$5
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/resnet50-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
 
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  export PATH=${WORKSPACE}/miniconda3/bin:$PATH
-  source ${WORKSPACE}/pytorch_setup/setvars.sh
-  source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
-  conda activate pytorch
+  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
+  #source ${WORKSPACE}/pytorch_setup/setvars.sh
+  #source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+  #conda activate pytorch
 fi
 
 # Run script
-OUTPUT_DIR=${OUTPUT_DIR} BATCH_SIZE=${BATCH_SIZE} DATASET_DIR=${DATASET} ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/${PRECISION}/${SCRIPT}
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/${PRECISION}/${SCRIPT}
diff --git a/tests/cicd/PyTorch/setup.sh b/tests/cicd/PyTorch/setup.sh
index ac036a4e7..70e10a761 100755
--- a/tests/cicd/PyTorch/setup.sh
+++ b/tests/cicd/PyTorch/setup.sh
@@ -9,16 +9,6 @@ WORKSPACE=$3
 AIKIT_RELEASE=$4
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  if [ ! -d "${WORKSPACE}/pytorch_setup" ]; then
-    mkdir -p ${WORKSPACE}/pytorch_setup
-    cd ${WORKSPACE}/oneapi_drop_tool
-    git submodule update --init --remote --recursive
-    python -m pip install -r requirements.txt
-    python pdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product ipytorch --release ${AIKIT_RELEASE} --part webimage --package-os linux --download-dir ${WORKSPACE}/pytorch_setup
-    cd ${WORKSPACE}/pytorch_setup
-    chmod +x webimage/l_ipytorch_*offline.sh
-    ./webimage/l_ipytorch_*_offline.sh -s -a --install-dir ${WORKSPACE}/pytorch_setup/ --silent --eula accept
-  fi
   if [ ! -d "${WORKSPACE}/miniconda3" ]; then
     cd ${WORKSPACE}
     curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
@@ -26,6 +16,17 @@ if [[ "${is_lkg_drop}" == "true" ]]; then
     chmod +x Miniconda3-latest-Linux-x86_64.sh
     ./Miniconda3-latest-Linux-x86_64.sh -b -f -p miniconda3
   fi
+  rm -rf ${WORKSPACE}/pytorch_setup
+  if [ ! -d "${WORKSPACE}/pytorch_setup" ]; then
+    mkdir -p ${WORKSPACE}/pytorch_setup
+    cd ${WORKSPACE}/oneapi_drop_tool
+    git submodule update --init --remote --recursive
+    python -m pip install -r requirements.txt
+    python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product ipytorch --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${WORKSPACE}/pytorch_setup
+    cd ${WORKSPACE}/pytorch_setup
+    chmod +x IPEX_installer-2024.0.0.*
+    ./IPEX_installer-2024.0.0.* -b -u -p ${WORKSPACE}/pytorch_setup
+  fi
 else
   pip install --upgrade pip
   echo "Installing pytorch"
diff --git a/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh b/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
index 1d7572da8..dc9a8623a 100755
--- a/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
+++ b/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
@@ -5,26 +5,28 @@ echo "Setup PyTorch Test Enviroment for SSD-ResNet34 Inference"
 
 PRECISION=$1
 SCRIPT=$2
-DATASET=$3
-OUTPUT_DIR=${OUTPUT_DIR-"tests/cicd/PyTorch/output/ssd-resnet34-inference/${PRECISION}"}
-WORKSPACE=$4
-is_lkg_drop=$5
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/ssd-resnet34-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
 
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  export PATH=${WORKSPACE}/miniconda3/bin:$PATH
-  source ${WORKSPACE}/pytorch_setup/setvars.sh
-  source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
-  source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
-  conda activate pytorch
+  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
+  #source ${WORKSPACE}/pytorch_setup/setvars.sh
+  #source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
+  #source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+  #conda activate pytorch
 fi
 
 # run model specific dependencies:
-pip install matplotlib Pillow pycocotools defusedxml
+pip install matplotlib Pillow defusedxml
+pip install pycocotools-fix
 
 # Download Pretrained Model:
 export CHECKPOINT_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
@@ -35,4 +37,4 @@ export DATASET_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
 ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/download_dataset.sh
 
 # Run script
-OUTPUT_DIR=${OUTPUT_DIR} BATCH_SIZE=${BATCH_SIZE} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET_DIR} ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/${SCRIPT} ${PRECISION}
+OUTPUT_DIR=${OUTPUT_DIR} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET_DIR} ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/${SCRIPT} ${PRECISION}
diff --git a/tests/cicd/TensorFlow/mobilenetv1-inference/run_test.sh b/tests/cicd/TensorFlow/mobilenetv1-inference/run_test.sh
new file mode 100755
index 000000000..d1050575f
--- /dev/null
+++ b/tests/cicd/TensorFlow/mobilenetv1-inference/run_test.sh
@@ -0,0 +1,33 @@
+ #!/bin/bash
+set -e
+
+echo "Setup TensorFlow Test Enviroment for MobileNet v1 Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/TensorFlow/mobilenetv1-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${PRECISION}" == "fp32" || "${PRECISION}" == "bfloat32" ]]; then
+  PRETRAINED_MODEL="/tf_dataset/pre-trained-models/mobilenet_v1/fp32/mobilenetv1_fp32_pretrained_model_new.pb"
+elif [[ "${PRECISION}" == "int8" ]]; then
+  PRETRAINED_MODEL="/tf_dataset/pre-trained-models/mobilenet_v1/int8/mobilenetv1_int8_pretrained_model.pb"
+fi
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
+  #source ${WORKSPACE}/tensorflow_setup/setvars.sh
+  #source ${WORKSPACE}/tensorflow_setup/compiler/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/mkl/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/tbb/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/mpi/latest/env/vars.sh
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+fi
+
+# run following script
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} PRETRAINED_MODEL=${PRETRAINED_MODEL} DATASET_DIR=${DATASET} ./quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/TensorFlow/resnet50v1.5-inference/run_test.sh b/tests/cicd/TensorFlow/resnet50v1.5-inference/run_test.sh
index 32d0b6475..428c10e45 100755
--- a/tests/cicd/TensorFlow/resnet50v1.5-inference/run_test.sh
+++ b/tests/cicd/TensorFlow/resnet50v1.5-inference/run_test.sh
@@ -5,10 +5,10 @@ echo "Setup TensorFlow Test Enviroment for ResNet50 v1.5 Inference"
 
 PRECISION=$1
 SCRIPT=$2
-DATASET=$3
-OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/TensorFlow/output/resnet50v1.5-inference/${SCRIPT}/${PRECISION}"}
-WORKSPACE=$4
-is_lkg_drop=$5
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/TensorFlow/resnet50v1.5-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
 
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}
@@ -22,14 +22,15 @@ elif [[ "${PRECISION}" == "int8" ]]; then
 fi
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  export PATH=${WORKSPACE}/miniconda3/bin:$PATH
-  source ${WORKSPACE}/tensorflow_setup/setvars.sh
-  source ${WORKSPACE}/tensorflow_setup/compiler/latest/env/vars.sh
-  source ${WORKSPACE}/tensorflow_setup/mkl/latest/env/vars.sh
-  source ${WORKSPACE}/tensorflow_setup/tbb/latest/env/vars.sh
-  source ${WORKSPACE}/tensorflow_setup/mpi/latest/env/vars.sh
-  conda activate tensorflow
+  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
+  #source ${WORKSPACE}/tensorflow_setup/setvars.sh
+  #source ${WORKSPACE}/tensorflow_setup/compiler/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/mkl/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/tbb/latest/env/vars.sh
+  #source ${WORKSPACE}/tensorflow_setup/mpi/latest/env/vars.sh
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+  #conda activate tensorflow
 fi
 
 # run following script
-OUTPUT_DIR=${OUTPUT_DIR} BATCH_SIZE=${BATCH_SIZE} PRECISION=${PRECISION} PRETRAINED_MODEL=${PRETRAINED_MODEL} DATASET_DIR=${DATASET} ./quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/${SCRIPT}
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} PRETRAINED_MODEL=${PRETRAINED_MODEL} DATASET_DIR=${DATASET} ./quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/TensorFlow/setup.sh b/tests/cicd/TensorFlow/setup.sh
index e3759790a..f7423d937 100755
--- a/tests/cicd/TensorFlow/setup.sh
+++ b/tests/cicd/TensorFlow/setup.sh
@@ -9,16 +9,6 @@ WORKSPACE=$3
 AIKIT_RELEASE=$4
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  if [ ! -d "${WORKSPACE}/tensorflow_setup" ]; then
-    mkdir -p ${WORKSPACE}/tensorflow_setup
-    cd ${WORKSPACE}/oneapi_drop_tool
-    git submodule update --init --remote --recursive
-    python -m pip install -r requirements.txt
-    python pdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product tensorflow --release ${AIKIT_RELEASE} --part webimage --package-os linux --download-dir ${WORKSPACE}/tensorflow_setup
-    cd ${WORKSPACE}/tensorflow_setup
-    chmod +x webimage/l_tensorflow_*offline.sh
-    ./webimage/l_tensorflow_*_offline.sh -s -a --install-dir ${WORKSPACE}/tensorflow_setup/ --silent --eula accept
-  fi
   if [ ! -d "${WORKSPACE}/miniconda3" ]; then
     cd ${WORKSPACE}
     curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
@@ -26,6 +16,17 @@ if [[ "${is_lkg_drop}" == "true" ]]; then
     chmod +x Miniconda3-latest-Linux-x86_64.sh
     ./Miniconda3-latest-Linux-x86_64.sh -b -f -p miniconda3
   fi
+  rm -rf ${WORKSPACE}/tensorflow_setup
+  if [ ! -d "${WORKSPACE}/tensorflow_setup" ]; then
+    mkdir -p ${WORKSPACE}/tensorflow_setup
+    cd ${WORKSPACE}/oneapi_drop_tool
+    git submodule update --init --remote --recursive
+    python -m pip install -r requirements.txt
+    python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product tensorflow --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${WORKSPACE}/tensorflow_setup
+    cd ${WORKSPACE}/tensorflow_setup
+    chmod +x ITEX_installer-2024.0.0.*
+    ./ITEX_installer-2024.0.0.* -b -u -p ${WORKSPACE}/tensorflow_setup
+  fi
 else
   pip install --upgrade pip
   echo "Installing tensorflow"
diff --git a/tests/cicd/mobilenetv1-inference/build.sh b/tests/cicd/mobilenetv1-inference/build.sh
deleted file mode 100755
index 0284c69e0..000000000
--- a/tests/cicd/mobilenetv1-inference/build.sh
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/bin/bash
-set -e
-
-echo "Setup TensorFlow Test Enviroment for MobileNet v1 Inference"
-
-BATCH_SIZE=$1
-PRECISION=$2
-BENCHMARK_TYPE=$3
-MODEL_DIR=${MODEL_DIR-"$(pwd)"}
-OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/output/${PRECISION}"}
-
-echo "Setup TensorFlow Test Enviroment..."
-# env setup
-VENV_DIR="${MODEL_ZOO_REPO_ROOT}/venv"
-if [ ! -d "${VENV_DIR}" ]; then
-  python3 -m venv ${VENV_DIR}
-fi
-source ${VENV_DIR}/bin/activate
-
-${VENV_DIR}/bin/python3 -m pip install --upgrade pip
-echo "Installing tensorflow"
-pip install intel-tensorflow==$4
-
-if [[ $2 == "fp32" ]];then
-  PRETRAINED_MODEL="/tf_dataset/pre-trained-models/mobilenet_v1/fp32/mobilenetv1_fp32_pretrained_model_new.pb"
-elif [ "$2" = "int8" ]; then
-  PRETRAINED_MODEL="/tf_dataset/pre-trained-models/mobilenet_v1/int8/mobilenetv1_int8_pretrained_model_new.pb"
-elif [ "$2" = "bfloat16" ];then
-  PRETRAINED_MODEL="/tf_dataset/pre-trained-models/mobilenet_v1/fp32/mobilenetv1_fp32_pretrained_model_new.pb"
-else
-  echo "No pretrained model is available for the specificied precision $2"
-fi
-
-echo "Running MobileNet V1 ${BENCHMARK_TYPE} for ${PRECISION} precision ..."
-# run following script
-cd ../../..
-OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} PRETRAINED_MODEL=${PRETRAINED_MODEL} ./quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/$3
-cd -
-
-deactivate
-rm -r ${VENV_DIR}
diff --git a/tests/cicd/resnet50v1.5-inference/build.sh b/tests/cicd/resnet50v1.5-inference/build.sh
deleted file mode 100755
index 93baf817d..000000000
--- a/tests/cicd/resnet50v1.5-inference/build.sh
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/bin/bash
-set -e
-
-echo "Setup TensorFlow Test Enviroment for ResNet50 v1.5 Inference"
-
-BATCH_SIZE=$1
-PRECISION=$2
-BENCHMARK_TYPE=$3
-MODEL_DIR=${MODEL_DIR-"$(pwd)"}
-OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/output/${PRECISION}"}
-PRETRAINED_MODEL=${PRETRAINED_MODEL-"/tf_dataset/pre-trained-models/resnet50v1_5/"${PRECISION}"/resnet50_v1.pb"}
-
-# env setup
-VENV_DIR="${MODEL_DIR}/venv"
-if [ ! -d "${VENV_DIR}" ]; then
-  python3 -m venv ${VENV_DIR}
-fi
-source ${VENV_DIR}/bin/activate
-
-${VENV_DIR}/bin/python3 -m pip install --upgrade pip
-echo "Installing tensorflow"
-pip install intel-tensorflow==$4
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "Downloading the pretrianed model..."
-  wget -O resnet50_v1.pb https://zenodo.org/record/2535873/files/resnet50_v1.pb
-  PRETRAINED_MODEL=$(pwd)"/model"
-  mv resnet50_v1.pb ${PRETRAINED_MODEL}
-  PRETRAINED_MODEL=${PRETRAINED_MODEL-"resnet50_v1.pb"}
-fi
-
-# run following script
-cd ../../..
-OUTPUT_DIR=${OUTPUT_DIR} BATCH_SIZE=${BATCH_SIZE} PRECISION=${PRECISION} PRETRAINED_MODEL=${PRETRAINED_MODEL} ./quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/${BENCHMARK_TYPE}
-cd -
-
-deactivate
-rm -r ${VENV_DIR}
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_bert_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_bert_args.json
index 379eb2a20..e434e8181 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_bert_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_bert_args.json
@@ -35,6 +35,24 @@
     "output": "python /workspace/intelai_models/inference/run_squad.py --init_checkpoint=/checkpoints/model.ckpt-3649 --vocab_file=/dataset/vocab.txt --bert_config_file=/dataset/bert_config.json --predict_file=/dataset/dev-v1.1.json --precision=fp32 --output_dir=/workspace/logs --predict_batch_size=128 --experimental_gelu=False --optimized_softmax=True --input_graph=/in_graph/bert.pb --do_predict=True  --mode=accuracy --inter_op_parallelism_threads=2 --intra_op_parallelism_threads=56 --warmup_steps=10 --steps=30 --num_cores_per_socket=28",
     "cpuset": "0-111"
   },
+  {
+    "_comment": "bert_large_int8_squad_inference",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=int8 --mode=inference --benchmark-dir=/workspace/benchmarks --checkpoint=/checkpoints --intelai-models=/workspace/intelai_models --in-graph=/in_graph/bert.pb --output-dir=/workspace/logs --batch-size=32 --data-location=/dataset --infer-option=SQuAD",
+    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 python /workspace/intelai_models/inference/run_squad.py --init_checkpoint=/checkpoints/model.ckpt-3649 --vocab_file=/dataset/vocab.txt --bert_config_file=/dataset/bert_config.json --predict_file=/dataset/dev-v1.1.json --precision=int8 --output_dir=/workspace/logs --predict_batch_size=32 --experimental_gelu=False --optimized_softmax=True --input_graph=/in_graph/bert.pb --do_predict=True  --inter_op_parallelism_threads=2 --intra_op_parallelism_threads=56 --warmup_steps=10 --steps=30 --num_cores_per_socket=28",
+    "cpuset": "0-111"
+  },
+  {
+    "_comment": "bert_large_int8_inference_optional_args",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=int8 --mode=inference --benchmark-dir=/workspace/benchmarks --checkpoint=/checkpoints --intelai-models=/workspace/intelai_models --in-graph=/in_graph/bert.pb --output-dir=/workspace/logs --data-location=/dataset --infer-option=SQuAD --num-intra-threads=28 --num-inter-threads=1 --benchmark-only --doc-stride=128 --max-seq-length=384",
+    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 python /workspace/intelai_models/inference/run_squad.py --init_checkpoint=/checkpoints/model.ckpt-3649 --vocab_file=/dataset/vocab.txt --bert_config_file=/dataset/bert_config.json --predict_file=/dataset/dev-v1.1.json --precision=int8 --output_dir=/workspace/logs --predict_batch_size=32 --experimental_gelu=False --optimized_softmax=True --input_graph=/in_graph/bert.pb --do_predict=True  --mode=benchmark --doc_stride=128 --max_seq_length=384 --inter_op_parallelism_threads=1 --intra_op_parallelism_threads=28 --warmup_steps=10 --steps=30 --num_cores_per_socket=28",
+    "cpuset": "0-111"
+  },
+  {
+    "_comment": "bert_large_int8_squad_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=int8 --mode=inference --benchmark-dir=/workspace/benchmarks --checkpoint=/checkpoints --intelai-models=/workspace/intelai_models --in-graph=/in_graph/bert.pb --output-dir=/workspace/logs --batch-size=32 --data-location=/dataset --infer-option=SQuAD --accuracy-only",
+    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 python /workspace/intelai_models/inference/run_squad.py --init_checkpoint=/checkpoints/model.ckpt-3649 --vocab_file=/dataset/vocab.txt --bert_config_file=/dataset/bert_config.json --predict_file=/dataset/dev-v1.1.json --precision=int8 --output_dir=/workspace/logs --predict_batch_size=32 --experimental_gelu=False --optimized_softmax=True --input_graph=/in_graph/bert.pb --do_predict=True  --mode=accuracy --inter_op_parallelism_threads=2 --intra_op_parallelism_threads=56 --warmup_steps=10 --steps=30 --num_cores_per_socket=28",
+    "cpuset": "0-111"
+  },
   {
     "_comment": "bert_large_bfloat16_squad_inference",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=bfloat16 --mode=inference --benchmark-dir=/workspace/benchmarks --checkpoint=/checkpoints --intelai-models=/workspace/intelai_models --batch-size=32 --output-dir=/workspace/logs --infer-option=SQuAD --data-location=/dataset --benchmark-only",
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_bert_large_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_bert_large_args.json
index 9c526288e..e5a00e859 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_bert_large_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_bert_large_args.json
@@ -12,11 +12,11 @@
   {
     "_comment": "bert_large_fp32_training_gpu",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=fp32 --mode=training --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=-1 --output-dir=/workspace/benchmarks/common/tensorflow/logs --num-train-steps=1 --benchmark-only --gpu --disable-tcmalloc=True --train-option=SQuAD --init-checkpoint=/checkpoints/bert_model.ckpt --vocab-file=/datasets/vocab.txt --config-file=/datasets/bert_config.json --do-predict=True --predict-file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json --do-train=True --train-file=/datasets/SQuAD-v1.1/train-v1.1.json --num-train-epochs=2 --learning-rate=3e-5 --max-seq-length=384 --doc-stride=128 --do-lower-case=True --experimental-gelu=False --optimized-softmax=True",
-    "output": "python /workspace/intelai_models/training/fp32/run_squad.py \\\n --output_dir=/workspace/benchmarks/common/tensorflow/logs \\\n --bert_config_file=/datasets/bert_config.json \\\n --do_train=True \\\n --train_batch_size=1 \\\n --accum_steps=1 \\\n --learning_rate=3e-05 \\\n --max_seq_length=384 \\\n --use_tpu=False \\\n --precision=fp32 \\\n --intra_op_parallelism_threads=56 \\\n --inter_op_parallelism_threads=2 \\\n --profile=False \\\n --do_lower_case=True \\\n --experimental_gelu=False \\\n --mpi_workers_sync_gradients=False \\\n --vocab_file=/datasets/vocab.txt \\\n --train_file=/datasets/SQuAD-v1.1/train-v1.1.json \\\n --predict_file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json \\\n --do_predict=True \\\n --num_train_epochs=2.0 \\\n --init_checkpoint=/checkpoints/bert_model.ckpt \\\n --doc_stride=128"
+    "output": "python /workspace/intelai_models/training/fp32/run_squad.py \\\n --output_dir=/workspace/benchmarks/common/tensorflow/logs \\\n --bert_config_file=/datasets/bert_config.json \\\n --do_train=True \\\n --train_batch_size=1 \\\n --accum_steps=1 \\\n --learning_rate=3e-05 \\\n --max_seq_length=384 \\\n --use_tpu=False \\\n --precision=fp32 \\\n --intra_op_parallelism_threads=56 \\\n --inter_op_parallelism_threads=2 \\\n --profile=False \\\n --do_lower_case=True \\\n --experimental_gelu=False \\\n --mpi_workers_sync_gradients=False \\\n --gpu=True \\\n --vocab_file=/datasets/vocab.txt \\\n --train_file=/datasets/SQuAD-v1.1/train-v1.1.json \\\n --predict_file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json \\\n --do_predict=True \\\n --num_train_epochs=2.0 \\\n --init_checkpoint=/checkpoints/bert_model.ckpt \\\n --doc_stride=128"
   },
   {
     "_comment": "bert_large_bfloat16_training_gpu",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=bert_large --precision=bfloat16 --mode=training --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=-1 --output-dir=/workspace/benchmarks/common/tensorflow/logs --num-train-steps=1 --benchmark-only --gpu --disable-tcmalloc=True --train-option=SQuAD --init-checkpoint=/checkpoints/bert_model.ckpt --vocab-file=/datasets/vocab.txt --config-file=/datasets/bert_config.json --do-predict=True --predict-file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json --do-train=True --train-file=/datasets/SQuAD-v1.1/train-v1.1.json --num-train-epochs=2 --learning-rate=3e-5 --max-seq-length=384 --doc-stride=128 --do-lower-case=True --experimental-gelu=False --optimized-softmax=True",
-    "output": "python /workspace/intelai_models/training/bfloat16/run_squad.py \\\n --output_dir=/workspace/benchmarks/common/tensorflow/logs \\\n --bert_config_file=/datasets/bert_config.json \\\n --do_train=True \\\n --train_batch_size=1 \\\n --accum_steps=1 \\\n --learning_rate=3e-05 \\\n --max_seq_length=384 \\\n --use_tpu=False \\\n --precision=bfloat16 \\\n --intra_op_parallelism_threads=56 \\\n --inter_op_parallelism_threads=2 \\\n --profile=False \\\n --do_lower_case=True \\\n --experimental_gelu=False \\\n --optimized_softmax=True \\\n --mpi_workers_sync_gradients=False \\\n --vocab_file=/datasets/vocab.txt \\\n --train_file=/datasets/SQuAD-v1.1/train-v1.1.json \\\n --predict_file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json \\\n --do_predict=True \\\n --num_train_epochs=2.0 \\\n --init_checkpoint=/checkpoints/bert_model.ckpt \\\n --doc_stride=128"
+    "output": "python /workspace/intelai_models/training/bfloat16/run_squad.py \\\n --output_dir=/workspace/benchmarks/common/tensorflow/logs \\\n --bert_config_file=/datasets/bert_config.json \\\n --do_train=True \\\n --train_batch_size=1 \\\n --accum_steps=1 \\\n --learning_rate=3e-05 \\\n --max_seq_length=384 \\\n --use_tpu=False \\\n --precision=bfloat16 \\\n --intra_op_parallelism_threads=56 \\\n --inter_op_parallelism_threads=2 \\\n --profile=False \\\n --do_lower_case=True \\\n --experimental_gelu=False \\\n --optimized_softmax=True \\\n --mpi_workers_sync_gradients=False \\\n --gpu=True \\\n --vocab_file=/datasets/vocab.txt \\\n --train_file=/datasets/SQuAD-v1.1/train-v1.1.json \\\n --predict_file=/datasets/SQuAD-v1.1/mini-dev-v1.1.json \\\n --do_predict=True \\\n --num_train_epochs=2.0 \\\n --init_checkpoint=/checkpoints/bert_model.ckpt \\\n --doc_stride=128"
   }
 ]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json
new file mode 100644
index 000000000..ae31914d6
--- /dev/null
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json
@@ -0,0 +1,17 @@
+[
+    { "_comment": "gpt_j_fp32_accuracy",
+      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=fp32 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_fp32_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
+      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision fp32 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_fp32_bs100_Accuracy_instance2_accuracy",
+      "cpuset": "0-111"},
+  
+    { "_comment": "gpt_j_bf16_accuracy",
+      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=bfloat16 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_bfloat16_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
+      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision bfloat16 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_bfloat16_bs100_Accuracy_instance2_accuracy",
+      "cpuset": "0-111"},
+  
+    { "_comment": "gpt_j_fp16_accuracy",
+      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=fp16 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_fp16_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
+      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision fp16 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_fp16_bs100_Accuracy_instance2_accuracy",
+      "cpuset": "0-111"}
+  
+  ]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json
new file mode 100644
index 000000000..cd892d446
--- /dev/null
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json
@@ -0,0 +1,31 @@
+[
+  { "_comment": "graphsage_fp32_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=fp32 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
+    "cpuset": "0-111"},
+
+  { "_comment": "graphsage_inference_benchmark_fp32",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp32 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
+    "cpuset": "0-111"},
+
+  { "_comment": "graphsage_bfloat16_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=bfloat16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
+    "cpuset": "0-111"},
+
+  { "_comment": "graphsage_inference_benchmark_bfloat16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision bfloat16 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
+    "cpuset": "0-111"},
+
+  { "_comment": "graphsage_fp16_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=fp16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
+    "cpuset": "0-111"},
+
+  { "_comment": "graphsage_inference_benchmark_fp16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp16 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
+    "cpuset": "0-111"}
+]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json
new file mode 100644
index 000000000..aa0175426
--- /dev/null
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json
@@ -0,0 +1,46 @@
+[
+  { "_comment": "mmoe_fp32_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=fp32 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
+    "cpuset": "0-111"},
+
+  { "_comment": "mmoe_inference_benchmark_fp32",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision fp32 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=fp32 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "mmoe_bf16_accuracy",
+  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=bfloat16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=bfloat16 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
+  "cpuset": "0-111"},
+
+  { "_comment": "mmoe_inference_benchmark_bf16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision bfloat16 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=bfloat16 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "mmoe_fp16_accuracy",
+  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=fp16 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
+  "cpuset": "0-111"},
+
+  { "_comment": "mmoe_inference_benchmark_fp16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision fp16 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=fp16 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "mmoe_training_fp32",
+  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp32 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
+  "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=fp32 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
+  "cpuset": "0-111"},
+
+  { "_comment": "mmoe_training_bfloat16",
+  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=bfloat16 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
+  "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=bfloat16 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
+  "cpuset": "0-111"},
+
+  { "_comment": "mmoe_training_fp16",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp16 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
+    "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=fp16 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
+    "cpuset": "0-111"}
+]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json
new file mode 100644
index 000000000..2fce8546a
--- /dev/null
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json
@@ -0,0 +1,31 @@
+[
+  { "_comment": "rgat_fp32_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=fp32 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "rgat_inference_benchmark_fp32",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp32 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
+    "cpuset": "0-111"},
+
+    { "_comment": "rgat_bfloat16_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=bfloat16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "rgat_inference_benchmark_bfloat16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision bfloat16 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
+    "cpuset": "0-111"},
+
+    { "_comment": "rgat_fp16_accuracy",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=fp16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
+    "cpuset": "0-111"},
+
+  { "_comment": "rgat_inference_benchmark_fp16",
+    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp16 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
+    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
+    "cpuset": "0-111"}
+]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json
new file mode 100644
index 000000000..d82b7ae03
--- /dev/null
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json
@@ -0,0 +1,16 @@
+[
+  { "_comment": "stable_diffusion_inference_benchmark_fp32",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp32 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp32 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "cpuset": "0-111"},
+
+  { "_comment": "stable_diffusion_inference_benchmark_bf16",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision bfloat16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=bfloat16 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "cpuset": "0-111"},
+
+  { "_comment": "stable_diffusion_inference_benchmark_fp16",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp16 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "cpuset": "0-111"}
+]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json
index 16fd11a0f..3cc658647 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json
@@ -27,6 +27,21 @@
     { "_comment": "vision_transformer_inference_benchmark_fp16",
       "input": "run_tf_benchmark.py --framework tensorflow --use-case image_recognition --precision fp16 --mode inference --model-name vision_transformer --batch-size 32 --in-graph /saved_model.pb --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
       "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp16/eval_image_classifier_inference.py --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=32 --warmup-steps=10 --steps=50",
-      "cpuset": "0-111"}
+      "cpuset": "0-111"},
+
+    { "_comment": "vit_training_fp32",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=fp32 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
+    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=fp32",
+    "cpuset": "0-111"},
+
+    { "_comment": "vit_training_fp16",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=fp16 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
+    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=fp16",
+    "cpuset": "0-111"},
+
+    { "_comment": "vit_training_bfloat16",
+    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=bfloat16 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
+    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=bfloat16",
+    "cpuset": "0-111"}
   
-  ]
+]
diff --git a/tests/unit/test_launch_benchmark.py b/tests/unit/test_launch_benchmark.py
index f3cf2aa53..a5fffba31 100644
--- a/tests/unit/test_launch_benchmark.py
+++ b/tests/unit/test_launch_benchmark.py
@@ -346,6 +346,18 @@ def test_disable_tcmalloc(launch_benchmark, mock_popen,
     assert "--env DISABLE_TCMALLOC=".format(expected_disable_tcmalloc) in docker_run_cmd
 
 
+@pytest.mark.parametrize("expected_onednn_graph", [["False"]])
+def test_onednn_graph(launch_benchmark, mock_popen,
+                      platform_mock, expected_onednn_graph):
+    platform_mock.return_value = platform_config.OS_TYPE
+    launch_benchmark.main()
+    assert mock_popen.called
+    args, _ = mock_popen.call_args
+    # convert the run command args to a string and then check for the custom volume mounts
+    docker_run_cmd = " ".join(args[0])
+    assert "--env ONEDNN_GRAPH=".format(expected_onednn_graph) in docker_run_cmd
+
+
 @pytest.mark.parametrize("numa_cores_per_instance_arg,socket_id_args,num_cores_arg,mpi_num_proc_arg,run_privileged",
                          [["4", -1, -1, None, True],
                           [None, -1, -1, None, False],
diff --git a/tools/.gitignore b/tools/.gitignore
deleted file mode 100644
index ea1472ec1..000000000
--- a/tools/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-output/
diff --git a/tools/ModelBuilderAdvanced.md b/tools/ModelBuilderAdvanced.md
deleted file mode 100644
index 338d64e56..000000000
--- a/tools/ModelBuilderAdvanced.md
+++ /dev/null
@@ -1,239 +0,0 @@
-# Model Builder - Advanced
-
-This document provides details and information on more advanced usages
-of the [model-builder script](/tools/scripts/model-builder). For
-basic usage instructions, see the [README.md](README.md) file.
-
-## Global Options
-
-Running `model-builder options` or `model-builder --help` will show the global options
-
-```
--h --help --verbose --dry-run
-```
-
-### --verbose
-
-This global option will output the internal call to docker run as well as debug output from assembler.py (which is invoked within docker).
-
-### --dry-run
-
-This global option will output what model-builder will execute; it shows just the docker run calls that would occur.
-
-## Custom Settings
-
-There are environment variables that can be set prior to running the
-model-builder script in order provide custom settings for the model
-packages and containers.
-
-| Variable | Default value | Description |
-|----------|---------------|-------------|
-| `MODEL_PACKAGE_DIR` | `../output` | Directory where model package .tar.gz files are located |
-| `LOCAL_REPO` | `model-zoo` | Local images will be built as `${LOCAL_REPO}:tag`. Tags are defined by the spec yml files |
-| `TENSORFLOW_TAG` | `latest` | Tag of the intel-optimized-tensorflow image to use as the base for versioned containers |
-| `TAG_PREFIX` | `${TENSORFLOW_TAG}` | Prefix used for the image tags (typically this will be the TF version) |
-| `MODEL_WORKSPACE` | `/workspace` | Location where the model package will be extracted in the model container |
-| `IMAGE_LIST_FILE` | None | Specify a file path where the list of built image/tags will be written. This is used by automated build scripts. |
-| `TEST_DIR` | None | Specify the root directory for testing docker partials. See run\_tests\_path argument to assembler for more details |
-
-
-You can set these environment variables to customize the model-builder
-settings. For example:
-
-```
-MODEL_PACKAGE_DIR=/tmp/model_packages model-builder
-```
-`model-builder` supports `CentOS` or `Ubuntu` based TensorFlow tags. Use the environment variable `TENSORFLOW_TAG` to specify the tag for the base `intel-optimized-tensorflow` image.
-For example:
-```
-TENSORFLOW_TAG=centos-8 model-builder make mobilenet-v1-int8-inference
-```
-You can find the list of supported models on CentOS in `tools/docker/specs/centos`.
-## Documentation text replacement
-
-When `init-spec` is run, model spec yaml file's documentation section has
-a `text_replace` dictionary that defines keyword and value pairs that
-will be replaced when the final README.md is generated. The final README.md
-can be generated using either `model-builder generate-documentation <model>`
-or `model-builder make <model>`. The `text_replace` section is _optional_,
-and if it doesn't exist then no text replacement will happen when
-documentation is generated.
-
-By default, when `init-spec` is run, the following text replacement
-options will be defined in the model's spec yaml file:
-
-| Keyword | Value |
-|---------|-------|
-| `<model name>` | The model's name formatted to be written in sentences (like `ResNet50` or `SSD-MobileNet`) |
-| `<precision>` | The model's precision formatted to be written in sentences (like `FP32` or `Int8`) |
-| `<mode>` | The mode for the model package/container (`inference` or `training`)
-| `<use_case>` | The model's use case formatted as it is in the model zoo directory structure (like `image_recognition` or `object_detection`) |
-| `<model-precision-mode>` | The model spec name, which consists of the model name, precision, and mode, as it's formatted in file names (like `resnet50-fp32-inference`) |
-
-An example of what this looks like in the spec yaml is below:
-```
-    documentation:
-        ...
-        text_replace:
-            <mode>: inference
-            <model name>: SSD-ResNet34
-            <precision>: FP32
-            <use case>: object_detection
-            <package url>:
-            <package name>: ssd-resnet34-fp32-inference.tar.gz
-            <package dir>: ssd-resnet34-fp32-inference
-            <docker image>:
-```
-
-> Note: Please make sure to fill in the package url and docker image
-> once the they have been uploaded and pushed to a repo.
-
-After `init-spec` is run, these values can be changed (for example, if
-the `<model name>` is not formatted correctly).
-
-The [documentation fragments](/tools/docker/docs) use the keywords.
-For example, [title.md](/tools/docker/docs/title.md) has:
-```
-<!--- 0. Title -->
-# <model name> <precision> <mode>
-```
-
-When the documentation is generated, the text subsitution will happen and
-the generated README.md will have the values filled in:
-```
-<!--- 0. Title -->
-# SSD-ResNet34 FP32 inference
-```
-
-## Framework argument
-
-The `--framework <framework>` (or `-f`) flag to the `model-builder` script
-refers the names fo the folders in the [specs folder](/tools/docker/specs):
-```
-tools/docker/specs/
-├── k8s
-├── ml
-├── pytorch
-└── tensorflow
-```
-
-See a list of the available frameworks using:
-```
-$ model-builder frameworks
-k8s ml pytorch tensorflow
-```
-
-The model-builder script uses this value for the `--spec_dir` and `--framework`
-args when calling the [assembler.py](/tools/docker/assembler.py).
-
-The `--framework` (or `-f`) flag applies to the following model-builder subcommands:
-* build (e.g `model-builder build -f ml xgboost`)
-* generate-dockerfile (e.g. `model-builder generate-dockerfile -f ml`)
-* generate-documentation (e.g. `model-builder generate-documentation -f tensorflow`)
-* generate-deployment (e.g. `model-builder generate-deployment -f k8s`)
-* images (e.g `model-builder images -f pytorch` or `model-builder images -f ml`)
-* init-spec (e.g. `model-builder init-spec -f tensorflow inceptionv4-fp32-inference`)
-* make (e.g. `model-builder make -f pytorch pytorch-resnet50-bfloat16-inference`)
-* models (e.g. `model-builder models -f pytorch`)
-* package (e.g. `model-builder package -f pytorch pytorch-resnet50-bfloat16-inference`)
-* packages (e.g. `model-builder packages -f tensorflow`)
-* run-test-suite (e.g. `model-builder run-test-suite -c generate-dockerfile -f pytorch`)
-
-> If no `--framework <framework>` (or `-f`) flag is passed to the
-> `model-builder`, the default behavior will be to use `tensorflow`.
-
-## Under the hood of the subcommands
-
-### Building packages
-
-The model-builder command will build packages by calling docker run on the imz-tf-tools container passing
-in arguments to assembler.py. This internal call looks like the following:
-
-```
-docker run --rm -u 503:20 -v <path-to-models-repo>/tools/docker:/tf -v $PWD:/tf/models imz-tf-tools python3 assembler.py --release dockerfiles --build_packages --model_dir=models --output_dir=models/output
-```
-
-For single targets such as `bert-large-fp32-training` the model-builder adds an argument:
-
-```
---only_tags_matching=.*bert-large-fp32-training$
-```
-
-### Constructing Dockerfiles
-
-The model-builder command will construct Dockerfiles by calling docker run on the imz-tf-tools container passing
-in arguments to assembler.py. This internal call looks like the following:
-
-```
-docker run --rm -u 503:20 -v <path-to-models-repo>/tools/docker:/tf -v <path-to-models-repo>/dockerfiles:/tf/dockerfiles imz-tf-tools python3 assembler.py --release dockerfiles --construct_dockerfiles
-```
-
-For single targets such as `bert-large-fp32-training` the model-builder adds an argument:
-
-```
---only_tags_matching=.*bert-large-fp32-training$
-```
-
-### Building images
-
-The model-builder command will build images by calling docker run on the imz-tf-tools container passing
-in arguments to assembler.py. This internal call looks like the following:
-
-```
-docker run --rm -v <path-to-models-repo>/tools/docker:/tf -v /var/run/docker.sock:/var/run/docker.sock imz-tf-tools python3 assembler.py --arg _TAG_PREFIX=latest --arg http_proxy= --arg https_proxy= --arg TENSORFLOW_TAG=latest --arg PACKAGE_DIR=model_packages --arg MODEL_WORKSPACE=/workspace --repository model-zoo --release versioned --build_images --only_tags_matching=.*bert-large-fp32-training$ --quiet
-```
-
-For single targets such as `bert-large-fp32-training` the model-builder adds an argument:
-
-```
---only_tags_matching=.*bert-large-fp32-training$
-```
-
-### Generating k8 deployments
-
-The model-builder's generate-deployment subcommand will generate kubernetes deployments for that model using the model's k8 package. The subcommand will use env values from the model specification's runtime.env yaml array. The subcommand will look for runtime.env replacements under `${XDG_CONFIG_HOME:-$HOME/.config}/model-builder/specs/k8s`[^Ψ]. The first time `model-builder generate-deployment` is run, it will copy ./tools/docker/specs to `${XDG_CONFIG_HOME:-$HOME/.config}/model-builder/specs`. The user can then edit the k8 specification files to provide env values specific to their k8 deployment. For example the user would replace USER_ID with their [id](https://man7.org/linux/man-pages/man1/id.1.html). The subcommand algorithm is as follows:
-
-[^Ψ]: If $HOME is a nfs mount point then you should chose an alternate directory eg: `XDG_CONFIG_HOME=/tmp/$USER/config model-builder generate-deployment ...`.
-
-1. Read in all specification files under tools/docker/specs.
-2. Copy tools/docker/specs to `${XDG_CONFIG_HOME:-$HOME/.config}/model-builder/specs` if the directory does not exist.
-3. Read in all specification files under `${XDG_CONFIG_HOME:-$HOME/.config}/model-builder/specs`.
-4. Replace any specifications in 1 with specifications from 2.
-5. Extract the k8 package into a temporary directory.
-6. For each env in the model's specification under runtime, call `kustomize cfg set <temp-dir> env.name env.value`.
-7. Output the specific deployment under `deployments/<model>` by calling `kustomize build <temp-dir>`.
-
-### Running test-suites
-
-The model-builder's run-test-suit subcommand will generate tests for one or more subcommands where each subcommand can be given one or more release-groups and one or more models.
-The syntax is shown below:
-
-```
-model-builder run-test-suite --command [build|generate-dockerfile|generate-documentation|generate-deployment|package] --release-group <release-group> <model> ...
-```
-
-The run-test-suite subcommand generates test cases using syntax compatible with [bats-core](https://github.com/bats-core/bats-core). After generation, it calls bats providing the generated script. Several examples with output are shown below
-
-1. `model-builder run-test-suite --command generate-documentation --command generate-dockerfile --release-group versioned resnet50-fp32-inference resnet50-int8-inference`
-
-```
- ✓ validate generate-documentation for resnet50-fp32-inference in release-group versioned creates quickstart/image_recognition/tensorflow/resnet50/inference/fp32/README.md
- ✓ validate generate-documentation for resnet50-int8-inference in release-group versioned creates quickstart/image_recognition/tensorflow/resnet50/inference/int8/README.md
- ✓ validate generate-dockerfile for resnet50-fp32-inference in release-group versioned creates intel-tf-image-recognition-resnet50-fp32-inference.Dockerfile
- ✓ validate generate-dockerfile for resnet50-int8-inference in release-group versioned creates intel-tf-image-recognition-resnet50-int8-inference.Dockerfile
-
-4 tests, 0 failures
-```
-
-2. `model-builder run-test-suite -c package -f k8s`
-
-```
- ✓ validate package for bert-large-fp32-training-k8s in framework k8s creates bert-large-fp32-training-k8s.tar.gz
- ✓ validate package for resnet50v1-5-fp32-inference-k8s in framework k8s creates resnet50v1-5-fp32-inference-k8s.tar.gz
- ✓ validate package for resnet50v1-5-fp32-training-k8s in framework k8s creates resnet50v1-5-fp32-training-k8s.tar.gz
- ✓ validate package for rfcn-fp32-inference-k8s in framework k8s creates rfcn-fp32-inference-k8s.tar.gz
- ✓ validate package for wide-deep-large-ds-fp32-training-k8s in framework k8s creates wide-deep-large-ds-fp32-training-k8s.tar.gz
-
-5 tests, 0 failures
-```
-
diff --git a/tools/README.md b/tools/README.md
deleted file mode 100644
index c3247207c..000000000
--- a/tools/README.md
+++ /dev/null
@@ -1,165 +0,0 @@
-# Model Zoo packaging and containers
-
-The [model-builder script](/tools/scripts/model-builder) is used to
-build model packages and containers. It is intended for use by CI/CD systems,
-but it can also be used by engineers to facilitate adding new packages/containers
-and for local testing.
-
-## Model Builder Setup
-
-**Prerequisites:**
-* Docker (this is already a requirement for running Intel Model Zoo scenarios in Docker)
-* a clone of the [IntelAI/models repo](https://github.com/IntelAI/models) (`git clone https://github.com/IntelAI/models`)
-
-**Setup steps**
-
-1. You can either install model-builder to a directory in your PATH, update your PATH to include where model-builder is located 
-   within the model repository under tools/scripts or call model-builder using the complete path name. 
-   Model-builder will check to make sure the model repository directories are accessible.
-
-   Either:
-
-   ```
-   install tools/scripts/model-builder /usr/local/bin
-   ```
-
-   or
-
-   ```
-   export PATH=$PWD/tools/scripts:$PATH
-   ```
-
-   or
-
-   ```
-   export MODELS_REPO=$HOME/src/models
-   $MODELS_REPO/tools/scripts/model-builder
-   ```
-
-
-2. Navigate to the `models` repo to execute the `model-builder`. Use the
-   `--help` flag to see a list of all the options.
-   ```
-   cd <path-to-models-repo>
-   model-builder --help
-   ```
-
-   Common `model-builder` commands are:
-   * `model-builder frameworks`: See a list of available frameworks
-   * `model-builder models -f <framework>`: See a list the available models
-   * `model-builder make -f <framework> <model>`: Builds the model package tar file,
-     constructs the dockerfile, and builds the docker image for the
-     specified model. The model specified should be one of the models
-     listed from `model-builder models -f <framework>`.
-   * `model-builder build -f <framework> <model>`: Builds the docker
-     container for the specified model. If no `--framework` or `-f`
-     argument is provided, the model builder defaults to use the
-     the `tensorflow` framework.
-   * `model-builder init-spec -f <framework> <new model> <optional use case>`:
-     Creates a new spec file for the specified model. See the
-     [section below](#steps-to-add-a-new-model-spec) for full
-     instructions on using this command. If no `--framework` or `-f` arg
-     is provided, `tensorflow` will be used. Providing a `<use case>` is
-     optional and is only used in the case where the model does not already
-     exist in the repo, and you want the model-builder to create the
-     model's directories for you.
-   * `model-builder generate-deployment <model>`: Generates the k8 deployment files
-     under ./deployments/... . It will use env values from user specification files 
-     under `${XDG_CONFIG_HOME:-$HOME/.config}/model-builder/specs/k8s`.
-
-For details on the model-builder script see the
-[model builder advanced documentation](ModelBuilderAdvanced.md).
-
-## Steps to add a new model spec
-
-Each model has a [spec file](docker/specs) which includes information
-that is used to build model packages, construct dockerfiles, and build
-images.
-
-The `model-builder` can be used to initialize a new model spec file, which
-will include as much information that can be auto-generated. Manual
-updates may need to be made for partials required by the model,
-additional build args, and the list of model package files may need to
-be tweaked if the code organization in the model zoo repo is non-standard.
-It's assumed that the model which you are creating a spec for has already
-been added to the model zoo repo.
-
-Follow the [model builder setup above](#model-builder-setup) prior to
-using these steps.
-
-1. Use the following command to initialize a new spec file for your model.
-   If a pretrained model needs to be included in the model package
-   (this will be the case for most inference models), then set the
-   `MODEL_URL` path to the intel-optimized-tensorflow gcloud release
-   bucket URL for the pretrained model. Setting `MODEL_URL` can be
-   omitted if there is no pretrained model that needs to be included
-   in the model package.
-   ```
-   MODEL_URL=<gcloud url> model-builder init-spec -f <framework> <spec name> <optional use case>
-   ```
-
-   > Note that if no `--framework` or `-f` is specifed, the `tensorflow`
-   > framework will be used. The framework specified here will determine
-   > which [spec folder](/tools/docker/specs) the yml will be added to
-   > (for example: tensorflow, pytorch, or ml).
-
-   Example to init-spec for TensorFlow inception v4 FP32 inference:
-   ```
-    MODEL_URL=https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/inceptionv4_fp32_pretrained_model.pb model-builder init-spec inceptionv4-fp32-inference
-   ```
-
-   The spec name should be formatted like `modelname-precision-mode`
-   (for example: `rfcn-fp32-inference` or `resnet50v1-5-fp32-training`)
-   where the model name matches the name of a model in the
-   [model zoo repo](https://github.com/intelai/models)
-   except with underscores changed to dashes.
-
-   If the model does not already exist in the model zoo repo, you can
-   provide the <use case> arg, and you will be prompted with a question
-   asking if you'd like the model-builder to create the directories for
-   you like:
-   ```
-   $ model-builder init-spec -f pytorch inceptionv3-fp32-inference image_recognition
-
-   Auto-generating a new spec file for: inceptionv3-fp32-inference
-   Using use case 'image_recognition' if the model's directories don't already exist
-
-   The directory "models/models/image_recognition/pytorch/inceptionv3/inference/fp32" does not exist.
-   Do you want the model-builder to create it? [y/N]
-   ```
-
-   When the `model-builder init-spec` command is run, it will create a yaml
-   file in [`tools/docker/specs/<framework>` folder](docker/specs)
-   named `<spec name>_spec.yml`. If you have not already created
-   [quickstart scripts](https://github.com/IntelAI/models/tree/master/quickstart)
-   for your model, the `init-spec` command also adds a quickstart template
-   script for your model.
-
-   The `model-builder init-spec` command will also copy markdown snippets from `tools/docker/docs`
-   to `quickstart/<use_case>/tensorflow/<model>/<mode>/<device>/<precision>/.docs`. These snippets are generic and
-   should be updated to include model specific information.
-
-   > **Optional:** If additional library installs (like pip installs) or
-   > setup is needed to run your model, then manual updates will need to
-   > be made to define other partials. Also, if your model does not
-   > follow the standard model zoo directory structure, then you will
-   > need to update the list of `files` in the spec yaml to list any
-   > other files that are needed to run your model. See the
-   > [docker documentation](/tools/docker/README.md) for details.
-
-2. Test the new spec file by running the `model-builder make` command:
-   ```
-   model-builder make -f <framework> <spec name>
-   ```
-   After this command runs, you'll have:
-   * A model README.md in your `quickstart/<use_case>/tensorflow/<model>/<mode>/<device>/<precision>/`
-     that was built using markdown snippets from `quickstart/<use_case>/tensorflow/<model>/<mode>/<device>/<precision>/.docs`.
-   * A model package tar file your `<models repo>/output` directory
-     (this can be used for testing the model on bare metal).
-   * A model dockerfile in your `<models repo>/dockerfiles` directory.
-     *Commit this file to your branch along with your model code.*
-   * A local docker image (check `docker images` to see the list of
-     images on your machine. This image can be used to locally test
-     running the model in a container).
-
-3. When the model spec yaml is complete, create an MR that adds the new files to the repo.
diff --git a/tools/k8s/devops/argo/config-map.yaml b/tools/k8s/devops/argo/config-map.yaml
deleted file mode 100644
index bfbb34b2e..000000000
--- a/tools/k8s/devops/argo/config-map.yaml
+++ /dev/null
@@ -1,10 +0,0 @@
-# This file describes the config settings available in the workflow controller configmap
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: workflow-controller-configmap
-data:
-  config: |
-    # Specifies the container runtime interface to use (default: docker)
-    # must be one of: docker, kubelet, k8sapi, pns
-    containerRuntimeExecutor: k8sapi
diff --git a/tools/k8s/devops/argo/kustomization.yaml b/tools/k8s/devops/argo/kustomization.yaml
deleted file mode 100644
index 68166bfde..000000000
--- a/tools/k8s/devops/argo/kustomization.yaml
+++ /dev/null
@@ -1,5 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-- config-map.yaml
-namespace: argo
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/.helmignore b/tools/k8s/devops/helm-charts/tensorflow-serving/.helmignore
deleted file mode 100644
index 0e8a0eb36..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/.helmignore
+++ /dev/null
@@ -1,23 +0,0 @@
-# Patterns to ignore when building packages.
-# This supports shell glob matching, relative path matching, and
-# negation (prefixed with !). Only one pattern per line.
-.DS_Store
-# Common VCS dirs
-.git/
-.gitignore
-.bzr/
-.bzrignore
-.hg/
-.hgignore
-.svn/
-# Common backup files
-*.swp
-*.bak
-*.tmp
-*.orig
-*~
-# Various IDEs
-.project
-.idea/
-*.tmproj
-.vscode/
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/Chart.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/Chart.yaml
deleted file mode 100644
index cbd1cfeaa..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/Chart.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-apiVersion: v2
-name: tensorflow-serving
-description: A Helm chart for Kubernetes
-
-# A chart can be either an 'application' or a 'library' chart.
-#
-# Application charts are a collection of templates that can be packaged into versioned archives
-# to be deployed.
-#
-# Library charts provide useful utilities or functions for the chart developer. They're included as
-# a dependency of application charts to inject those utilities and functions into the rendering
-# pipeline. Library charts do not define any templates and therefore cannot be deployed.
-type: application
-
-# This is the chart version. This version number should be incremented each time you make changes
-# to the chart and its templates, including the app version.
-# Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.0
-
-# This is the version number of the application being deployed. This version number should be
-# incremented each time you make changes to the application. Versions are not expected to
-# follow Semantic Versioning. They should reflect the version the application is using.
-appVersion: 2.8.0
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/README.md b/tools/k8s/devops/helm-charts/tensorflow-serving/README.md
deleted file mode 100644
index 63517ab92..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/README.md
+++ /dev/null
@@ -1,57 +0,0 @@
-# A Helm chart for TensorFlow Serving
-
-Prior to running this how-to, please ensure you have `Helm 3` installed per instructions on this page:
-
-https://helm.sh/docs/intro/install/
-
-To install a Helm chart that deploys a TensorFlow Serving, try this:
-```shell
-# Note: Ensure a namespace with the following name exists:
-export SERVING_NAMESPACE=<NAME_SPACE_TO_DEPLOY_CHART>   # tensorflow-serving
-export CHART_NAME=<NAME_OF_DEPLOYD_CHART>   # resnet50v1-5
-# Navigate to the folder where the desired chart is located:
-cd tools/k8s/devops/helm-charts/tensorflow-serving
-
-# Install a Serving chart
-helm install \
-     --namespace $SERVING_NAMESPACE \
-     --debug $CHART_NAME . \
-     --set service.internalPort=8500 \
-     --set podSecurityContext.fsGroup=<YOUR_FS_GROUP> \
-     --set podSecurityContext.runAsGroup=<YOUR_DESIRED_GROUP> \
-     --set podSecurityContext.runAsUser=<YOUR_USER_ID> \
-     --set model_base_path=<MODEL_BASE_PATH_ENV_VALUE> \
-     --set model_name=<MODEL_NAME_ENV_VALUE> \
-     --set models_path=<MODEL_LOCAL_PATH> \
-     --set replicaCount=<SERVING_POD_REPLICAS>
-```
-
-This returns something like this:
-```shell
-NOTES:
-1. Get the application URL by running these commands:
-  export POD_NAME=$(kubectl get pods --namespace tensorflow-serving -l "app.kubernetes.io/name=tensorflow-serving,app.kubernetes.io/instance=resnet50v1-5" -o jsonpath="{.items[0].metadata.name}")
-  export CONTAINER_PORT=$(kubectl get pod --namespace tensorflow-serving $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
-  echo "Visit http://127.0.0.1:8501 to use your application"
-  kubectl --namespace tensorflow-serving port-forward $POD_NAME 8501:$CONTAINER_PORT
-```
-
-Once run the above commands prompted by NOTES, on another shell, try the following:
-
-```shell
-# Clone TensorFlow Serving repo:
-git clone https://github.com/tensorflow/serving.git
-export TF_SERVING_ROOT=$(pwd)/serving
-
-# Setup a virtual environment:
-python3 -m virtualenv -p python3 .venv3
-. .venv3/bin/activate
-pip install requests
-
-# Send rRPC request to your served model:
-python $TF_SERVING_ROOT/tensorflow_serving/example/resnet_client.py
-```
-This will return something like the following:
-```shell
-Prediction class: 286, avg latency: 57.318000000000005 ms
-```
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/NOTES.txt b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/NOTES.txt
deleted file mode 100644
index 528b5036c..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/NOTES.txt
+++ /dev/null
@@ -1,22 +0,0 @@
-1. Get the application URL by running these commands:
-{{- if .Values.ingress.enabled }}
-{{- range $host := .Values.ingress.hosts }}
-  {{- range .paths }}
-  http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
-  {{- end }}
-{{- end }}
-{{- else if contains "NodePort" .Values.service.type }}
-  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "tensorflow-serving.fullname" . }})
-  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
-  echo http://$NODE_IP:$NODE_PORT
-{{- else if contains "LoadBalancer" .Values.service.type }}
-     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
-           You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "tensorflow-serving.fullname" . }}'
-  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "tensorflow-serving.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
-  echo http://$SERVICE_IP:{{ .Values.service.port }}
-{{- else if contains "ClusterIP" .Values.service.type }}
-  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "tensorflow-serving.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
-  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
-  echo "Visit http://127.0.0.1:8501 to use your application"
-  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8501:$CONTAINER_PORT
-{{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/_helpers.tpl b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/_helpers.tpl
deleted file mode 100644
index 15a01ad1b..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/_helpers.tpl
+++ /dev/null
@@ -1,62 +0,0 @@
-{{/*
-Expand the name of the chart.
-*/}}
-{{- define "tensorflow-serving.name" -}}
-{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
-{{- end }}
-
-{{/*
-Create a default fully qualified app name.
-We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
-If release name contains chart name it will be used as a full name.
-*/}}
-{{- define "tensorflow-serving.fullname" -}}
-{{- if .Values.fullnameOverride }}
-{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
-{{- else }}
-{{- $name := default .Chart.Name .Values.nameOverride }}
-{{- if contains $name .Release.Name }}
-{{- .Release.Name | trunc 63 | trimSuffix "-" }}
-{{- else }}
-{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
-{{- end }}
-{{- end }}
-{{- end }}
-
-{{/*
-Create chart name and version as used by the chart label.
-*/}}
-{{- define "tensorflow-serving.chart" -}}
-{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
-{{- end }}
-
-{{/*
-Common labels
-*/}}
-{{- define "tensorflow-serving.labels" -}}
-helm.sh/chart: {{ include "tensorflow-serving.chart" . }}
-{{ include "tensorflow-serving.selectorLabels" . }}
-{{- if .Chart.AppVersion }}
-app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
-{{- end }}
-app.kubernetes.io/managed-by: {{ .Release.Service }}
-{{- end }}
-
-{{/*
-Selector labels
-*/}}
-{{- define "tensorflow-serving.selectorLabels" -}}
-app.kubernetes.io/name: {{ include "tensorflow-serving.name" . }}
-app.kubernetes.io/instance: {{ .Release.Name }}
-{{- end }}
-
-{{/*
-Create the name of the service account to use
-*/}}
-{{- define "tensorflow-serving.serviceAccountName" -}}
-{{- if .Values.serviceAccount.create }}
-{{- default (include "tensorflow-serving.fullname" .) .Values.serviceAccount.name }}
-{{- else }}
-{{- default "default" .Values.serviceAccount.name }}
-{{- end }}
-{{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/deployment.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/deployment.yaml
deleted file mode 100644
index caf8a1fed..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/deployment.yaml
+++ /dev/null
@@ -1,74 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: {{ include "tensorflow-serving.fullname" . }}
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-spec:
-  {{- if not .Values.autoscaling.enabled }}
-  replicas: {{ .Values.replicaCount }}
-  {{- end }}
-  selector:
-    matchLabels:
-      {{- include "tensorflow-serving.selectorLabels" . | nindent 6 }}
-  template:
-    metadata:
-      {{- with .Values.podAnnotations }}
-      annotations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      labels:
-        {{- include "tensorflow-serving.selectorLabels" . | nindent 8 }}
-    spec:
-      {{- with .Values.imagePullSecrets }}
-      imagePullSecrets:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      serviceAccountName: {{ include "tensorflow-serving.serviceAccountName" . }}
-      securityContext:
-        {{- toYaml .Values.podSecurityContext | nindent 8 }}
-      containers:
-        - name: {{ .Chart.Name }}
-          securityContext:
-            {{- toYaml .Values.securityContext | nindent 12 }}
-          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
-          imagePullPolicy: {{ .Values.image.pullPolicy }}
-          env:
-            - name: MODEL_NAME
-              value: {{ .Values.model_name }}
-            - name: MODEL_BASE_PATH
-              value : {{ .Values.model_base_path }}
-          ports:
-            - name: http
-              containerPort: 8501
-              protocol: TCP
-#          livenessProbe:
-#            httpGet:
-#              path: /
-#              port: http
-#          readinessProbe:
-#            httpGet:
-#              path: /
-#              port: http
-          resources:
-            {{- toYaml .Values.resources | nindent 12 }}
-          volumeMounts:
-            - name: models
-              mountPath: {{ .Values.models_path }}
-      volumes:
-        - name: models
-          hostPath:
-            path: {{ .Values.models_path }}
-
-      {{- with .Values.nodeSelector }}
-      nodeSelector:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      {{- with .Values.affinity }}
-      affinity:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      {{- with .Values.tolerations }}
-      tolerations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/hpa.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/hpa.yaml
deleted file mode 100644
index 03c5be688..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/hpa.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-{{- if .Values.autoscaling.enabled }}
-apiVersion: autoscaling/v2beta1
-kind: HorizontalPodAutoscaler
-metadata:
-  name: {{ include "tensorflow-serving.fullname" . }}
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-spec:
-  scaleTargetRef:
-    apiVersion: apps/v1
-    kind: Deployment
-    name: {{ include "tensorflow-serving.fullname" . }}
-  minReplicas: {{ .Values.autoscaling.minReplicas }}
-  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
-  metrics:
-    {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
-    - type: Resource
-      resource:
-        name: cpu
-        targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
-    {{- end }}
-    {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
-    - type: Resource
-      resource:
-        name: memory
-        targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
-    {{- end }}
-{{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/ingress.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/ingress.yaml
deleted file mode 100644
index f89a3d24b..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/ingress.yaml
+++ /dev/null
@@ -1,41 +0,0 @@
-{{- if .Values.ingress.enabled -}}
-{{- $fullName := include "tensorflow-serving.fullname" . -}}
-{{- $svcPort := .Values.service.port -}}
-{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
-apiVersion: networking.k8s.io/v1beta1
-{{- else -}}
-apiVersion: extensions/v1beta1
-{{- end }}
-kind: Ingress
-metadata:
-  name: {{ $fullName }}
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-  {{- with .Values.ingress.annotations }}
-  annotations:
-    {{- toYaml . | nindent 4 }}
-  {{- end }}
-spec:
-  {{- if .Values.ingress.tls }}
-  tls:
-    {{- range .Values.ingress.tls }}
-    - hosts:
-        {{- range .hosts }}
-        - {{ . | quote }}
-        {{- end }}
-      secretName: {{ .secretName }}
-    {{- end }}
-  {{- end }}
-  rules:
-    {{- range .Values.ingress.hosts }}
-    - host: {{ .host | quote }}
-      http:
-        paths:
-          {{- range .paths }}
-          - path: {{ .path }}
-            backend:
-              serviceName: {{ $fullName }}
-              servicePort: {{ $svcPort }}
-          {{- end }}
-    {{- end }}
-  {{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/service.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/service.yaml
deleted file mode 100644
index bccbad5f7..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/service.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: {{ include "tensorflow-serving.fullname" . }}
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-spec:
-  type: {{ .Values.service.type }}
-  ports:
-    - port: {{ .Values.service.port }}
-      targetPort: http
-      protocol: TCP
-      name: http
-  selector:
-    {{- include "tensorflow-serving.selectorLabels" . | nindent 4 }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/serviceaccount.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/serviceaccount.yaml
deleted file mode 100644
index 511f1ab10..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/serviceaccount.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-{{- if .Values.serviceAccount.create -}}
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: {{ include "tensorflow-serving.serviceAccountName" . }}
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-  {{- with .Values.serviceAccount.annotations }}
-  annotations:
-    {{- toYaml . | nindent 4 }}
-  {{- end }}
-{{- end }}
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/tests/test-connection.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/templates/tests/test-connection.yaml
deleted file mode 100644
index 2787ea64c..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/templates/tests/test-connection.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  name: "{{ include "tensorflow-serving.fullname" . }}-test-connection"
-  labels:
-    {{- include "tensorflow-serving.labels" . | nindent 4 }}
-  annotations:
-    "helm.sh/hook": test
-spec:
-  containers:
-    - name: wget
-      image: busybox
-      command: ['wget']
-      args: ['{{ include "tensorflow-serving.fullname" . }}:{{ .Values.service.port }}']
-  restartPolicy: Never
diff --git a/tools/k8s/devops/helm-charts/tensorflow-serving/values.yaml b/tools/k8s/devops/helm-charts/tensorflow-serving/values.yaml
deleted file mode 100644
index c152402f4..000000000
--- a/tools/k8s/devops/helm-charts/tensorflow-serving/values.yaml
+++ /dev/null
@@ -1,87 +0,0 @@
-# Default values for tensorflow-serving.
-# This is a YAML-formatted file.
-# Declare variables to be passed into your templates.
-
-replicaCount: 1
-
-image:
-  repository: intel/intel-optimized-tensorflow-serving
-  pullPolicy: IfNotPresent
-  # Overrides the image tag whose default is the chart appVersion.
-  tag: "latest"
-
-imagePullSecrets: []
-nameOverride: ""
-fullnameOverride: ""
-
-serviceAccount:
-  # Specifies whether a service account should be created
-  create: true
-  # Annotations to add to the service account
-  annotations: {}
-  # The name of the service account to use.
-  # If not set and create is true, a name is generated using the fullname template
-  name: ""
-
-podAnnotations: {}
-
-podSecurityContext:
-  fsGroup: 0
-  runAsGroup: 0
-  runAsUser: 0
-
-securityContext: {}
-  # capabilities:
-  #   drop:
-  #   - ALL
-  # readOnlyRootFilesystem: true
-  # runAsNonRoot: true
-  # runAsUser: 1000
-
-service:
-  internalPort: 8501
-  type: ClusterIP
-  port: 80
-
-ingress:
-  enabled: false
-  annotations: {}
-    # kubernetes.io/ingress.class: nginx
-    # kubernetes.io/tls-acme: "true"
-  hosts:
-    - host: chart-example.local
-      paths: []
-  tls: []
-  #  - secretName: chart-example-tls
-  #    hosts:
-  #      - chart-example.local
-
-resources: {}
-  # We usually recommend not to specify default resources and to leave this as a conscious
-  # choice for the user. This also increases chances charts run on environments with little
-  # resources, such as Minikube. If you do want to specify resources, uncomment the following
-  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
-  # limits:
-  #   cpu: 100m
-  #   memory: 128Mi
-  # requests:
-  #   cpu: 100m
-  #   memory: 128Mi
-
-autoscaling:
-  enabled: false
-  minReplicas: 1
-  maxReplicas: 100
-  targetCPUUtilizationPercentage: 80
-  # targetMemoryUtilizationPercentage: 80
-
-nodeSelector: {}
-
-tolerations: []
-
-affinity: {}
-
-model_base_path: ""
-model_name: ""
-models_path: ""
-model_port: 8500
diff --git a/tools/k8s/devops/operators/mpi-operator/README.md b/tools/k8s/devops/operators/mpi-operator/README.md
deleted file mode 100644
index 0f2364ed0..000000000
--- a/tools/k8s/devops/operators/mpi-operator/README.md
+++ /dev/null
@@ -1,55 +0,0 @@
-# mpi-operator
-
-Based on [mpi-operator](https://github.com/kubeflow/mpi-operator/tree/master/deploy/v1).
-Both images within the Deployment use tag v0.2.3.
-
-The latest tag for mpi-operator has problems creating workers. The error is '/bin/sh: /etc/hosts: Permission denied'. 
-This is due to a bug in mpi-operator in the 'latest' container image when the workers run as non-root. 
-See this [issue](https://github.com/kubeflow/mpi-operator/issues/288).<br/>
-
-## Install
-
-> kubectl apply -f mpi-operator.yaml
-
-## Delete
-
-> kubectl delete -f mpi-operator.yaml
-
-# mpi-operator-namespaced
-
-`mpi-operator-namespaced.yaml` is based on `mpi-operator.yaml` & helps address a different use case than the vanilla mpi-operator.
-
-This approach was born out of an issue we encountered with multiple mpi-operators in the same cluster.
-
-## Problem Statement:
-
-Consider two mpi-operators launched within the same cluster but in different namespaces.
-When an MPIJob is launched by any user on that cluster, both mpi-operators would listen 
-to any new MPIJob that has launched anywhere on the cluster.
-Because of this, an issue with mpi-operator leader election occurs, causing none of the mpi-operators to 
-take up the newly created MPIJob and hence failing to create any pods.
-
-## Solution:
-
-When mpi-operator launches the operator it starts with some default settings, 
-- it automatically scopes the operator to `mpi-operator` namespace. 
-- it also locks the operator to `mpi-operator` namespace.
-
-To override these defaults, `mpi-operator` provides two run-time container args that allow 
-each mpi-operator to be scoped to a particular namespace as opposed to a Cluster scope.
-
-```
---namespace=<your-namespace>
---lock-namespace=<your-namespace>
-```
-
-Adding these to the dynamic container args section would run the mpi-operator docker image with the namespace you specify.
-This will enable the mpi-operator to listen to the MPIJobs that originate only in the specified namespace and not cluster-wide.
-
-This solution eliminates the issue of leader election in a cluster scoped setting.
-
-## Uses:
-
-The recommended use of this configuration would be when the user would like to scope the 
-operator to a particular namespace & avoid conflicts with any other actively running
-mpi-operators on the cluster.
diff --git a/tools/k8s/devops/operators/mpi-operator/mpi-operator-namespaced.yaml b/tools/k8s/devops/operators/mpi-operator/mpi-operator-namespaced.yaml
deleted file mode 100644
index 737d13744..000000000
--- a/tools/k8s/devops/operators/mpi-operator/mpi-operator-namespaced.yaml
+++ /dev/null
@@ -1,196 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: <your-namespace>
----
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  name: mpijobs.kubeflow.org
-spec:
-  group: kubeflow.org
-  names:
-    kind: MPIJob
-    plural: mpijobs
-    shortNames:
-    - mj
-    - mpij
-    singular: mpijob
-  scope: Namespaced
-  subresources:
-    status: {}
-  validation:
-    openAPIV3Schema:
-      properties:
-        spec:
-          properties:
-            mpiReplicaSpecs:
-              properties:
-                Launcher:
-                  properties:
-                    replicas:
-                      maximum: 1
-                      minimum: 1
-                      type: integer
-                Worker:
-                  properties:
-                    replicas:
-                      minimum: 1
-                      type: integer
-            slotsPerWorker:
-              minimum: 1
-              type: integer
-  version: v1alpha2
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: mpi-operator
-  namespace: <your-namespace>
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: role
-  namespace: <your-namespace>
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - configmaps
-  - serviceaccounts
-  verbs:
-  - create
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  verbs:
-  - get
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - pods/exec
-  verbs:
-  - create
-- apiGroups:
-  - ""
-  resources:
-  - endpoints
-  verbs:
-  - create
-  - get
-  - update
-- apiGroups:
-  - ""
-  resources:
-  - events
-  verbs:
-  - create
-  - patch
-- apiGroups:
-  - rbac.authorization.k8s.io
-  resources:
-  - roles
-  - rolebindings
-  verbs:
-  - create
-  - list
-  - watch
-- apiGroups:
-  - policy
-  resources:
-  - poddisruptionbudgets
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - apps
-  resources:
-  - statefulsets
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - batch
-  resources:
-  - jobs
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - apiextensions.k8s.io
-  resources:
-  - customresourcedefinitions
-  verbs:
-  - create
-  - get
-- apiGroups:
-  - kubeflow.org
-  resources:
-  - mpijobs
-  - mpijobs/finalizers
-  - mpijobs/status
-  verbs:
-  - '*'
-- apiGroups:
-  - scheduling.incubator.k8s.io
-  - scheduling.sigs.dev
-  resources:
-  - queues
-  - podgroups
-  verbs:
-  - '*'
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: role-binding
-  namespace: <your-namespace>
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: role
-subjects:
-- kind: ServiceAccount
-  name: mpi-operator
-  namespace: <your-namespace>
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  labels:
-    app: mpi-operator
-  name: mpi-operator
-  namespace: <your-namespace>
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: mpi-operator
-  template:
-    metadata:
-      labels:
-        app: mpi-operator
-    spec:
-      containers:
-      - args:
-        - -alsologtostderr
-        - --kubectl-delivery-image
-        - mpioperator/kubectl-delivery:latest
-        - --namespace=<your-namespace>
-        - --lock-namespace=<your-namespace>
-        image: mpioperator/mpi-operator:v0.2.3
-        imagePullPolicy: Always
-        name: mpi-operator
-      serviceAccountName: mpi-operator
diff --git a/tools/k8s/devops/operators/mpi-operator/mpi-operator.yaml b/tools/k8s/devops/operators/mpi-operator/mpi-operator.yaml
deleted file mode 100644
index 23662b42b..000000000
--- a/tools/k8s/devops/operators/mpi-operator/mpi-operator.yaml
+++ /dev/null
@@ -1,192 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: mpi-operator
----
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  name: mpijobs.kubeflow.org
-spec:
-  group: kubeflow.org
-  names:
-    kind: MPIJob
-    plural: mpijobs
-    shortNames:
-    - mj
-    - mpij
-    singular: mpijob
-  scope: Namespaced
-  subresources:
-    status: {}
-  validation:
-    openAPIV3Schema:
-      properties:
-        spec:
-          properties:
-            mpiReplicaSpecs:
-              properties:
-                Launcher:
-                  properties:
-                    replicas:
-                      maximum: 1
-                      minimum: 1
-                      type: integer
-                Worker:
-                  properties:
-                    replicas:
-                      minimum: 1
-                      type: integer
-            slotsPerWorker:
-              minimum: 1
-              type: integer
-  version: v1alpha2
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: mpi-operator
-  namespace: mpi-operator
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: mpi-operator
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - configmaps
-  - serviceaccounts
-  verbs:
-  - create
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  verbs:
-  - get
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - pods/exec
-  verbs:
-  - create
-- apiGroups:
-  - ""
-  resources:
-  - endpoints
-  verbs:
-  - create
-  - get
-  - update
-- apiGroups:
-  - ""
-  resources:
-  - events
-  verbs:
-  - create
-  - patch
-- apiGroups:
-  - rbac.authorization.k8s.io
-  resources:
-  - roles
-  - rolebindings
-  verbs:
-  - create
-  - list
-  - watch
-- apiGroups:
-  - policy
-  resources:
-  - poddisruptionbudgets
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - apps
-  resources:
-  - statefulsets
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - batch
-  resources:
-  - jobs
-  verbs:
-  - create
-  - list
-  - update
-  - watch
-- apiGroups:
-  - apiextensions.k8s.io
-  resources:
-  - customresourcedefinitions
-  verbs:
-  - create
-  - get
-- apiGroups:
-  - kubeflow.org
-  resources:
-  - mpijobs
-  - mpijobs/finalizers
-  - mpijobs/status
-  verbs:
-  - '*'
-- apiGroups:
-  - scheduling.incubator.k8s.io
-  - scheduling.sigs.dev
-  resources:
-  - queues
-  - podgroups
-  verbs:
-  - '*'
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: mpi-operator
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: mpi-operator
-subjects:
-- kind: ServiceAccount
-  name: mpi-operator
-  namespace: mpi-operator
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  labels:
-    app: mpi-operator
-  name: mpi-operator
-  namespace: mpi-operator
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: mpi-operator
-  template:
-    metadata:
-      labels:
-        app: mpi-operator
-    spec:
-      containers:
-      - args:
-        - -alsologtostderr
-        - --kubectl-delivery-image
-        - mpioperator/kubectl-delivery:v0.2.3
-        image: mpioperator/mpi-operator:v0.2.3
-        imagePullPolicy: Always
-        name: mpi-operator
-      serviceAccountName: mpi-operator
diff --git a/tools/k8s/devops/operators/nfd-operator/README.md b/tools/k8s/devops/operators/nfd-operator/README.md
deleted file mode 100644
index 0fc8008a4..000000000
--- a/tools/k8s/devops/operators/nfd-operator/README.md
+++ /dev/null
@@ -1,12 +0,0 @@
-# nfd-operator
-
-Based on the operator hub. See [nfd-operator](https://operatorhub.io/operator/nfd-operator)
-
-## Install
-
-> kubectl apply -f nfd-operator.yaml
-
-## Delete
-
-> kubectl delete -f nfd-operator.yaml
-
diff --git a/tools/k8s/devops/operators/nfd-operator/nfd-operator.yaml b/tools/k8s/devops/operators/nfd-operator/nfd-operator.yaml
deleted file mode 100644
index e0bc7d2e9..000000000
--- a/tools/k8s/devops/operators/nfd-operator/nfd-operator.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-apiVersion: operators.coreos.com/v1alpha1
-kind: Subscription
-metadata:
-  name: my-nfd-operator
-  namespace: operators
-spec:
-  channel: stable
-  name: nfd-operator
-  source: operatorhubio-catalog
-  sourceNamespace: olm
-
diff --git a/tools/k8s/devops/operators/olm-operator/README.md b/tools/k8s/devops/operators/olm-operator/README.md
deleted file mode 100644
index 20a9135c0..000000000
--- a/tools/k8s/devops/operators/olm-operator/README.md
+++ /dev/null
@@ -1,14 +0,0 @@
-# olm-operator
-
-See [operator-lifecycle-manager](https://github.com/operator-framework/operator-lifecycle-manager/tree/master/deploy/chart)
-
-## Install
-
-> kubectl apply -f crds.yaml
-> kubectl apply -f olm.yaml
-
-## Delete
-
-> kubectl apply -f olm.yaml
-> kubectl apply -f crds.yaml
-
diff --git a/tools/k8s/devops/operators/olm-operator/crds.yaml b/tools/k8s/devops/operators/olm-operator/crds.yaml
deleted file mode 100644
index 78c15fe69..000000000
--- a/tools/k8s/devops/operators/olm-operator/crds.yaml
+++ /dev/null
@@ -1,11794 +0,0 @@
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: catalogsources.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: CatalogSource
-    listKind: CatalogSourceList
-    plural: catalogsources
-    shortNames:
-    - catsrc
-    singular: catalogsource
-  scope: Namespaced
-  versions:
-  - additionalPrinterColumns:
-    - description: The pretty name of the catalog
-      jsonPath: .spec.displayName
-      name: Display
-      type: string
-    - description: The type of the catalog
-      jsonPath: .spec.sourceType
-      name: Type
-      type: string
-    - description: The publisher of the catalog
-      jsonPath: .spec.publisher
-      name: Publisher
-      type: string
-    - jsonPath: .metadata.creationTimestamp
-      name: Age
-      type: date
-    name: v1alpha1
-    schema:
-      openAPIV3Schema:
-        description: CatalogSource is a repository of CSVs, CRDs, and operator packages.
-        type: object
-        required:
-        - metadata
-        - spec
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            type: object
-            required:
-            - sourceType
-            properties:
-              address:
-                description: 'Address is a host that OLM can use to connect to a pre-existing
-                  registry. Format: <registry-host or ip>:<port> Only used when SourceType
-                  = SourceTypeGrpc. Ignored when the Image field is set.'
-                type: string
-              configMap:
-                description: ConfigMap is the name of the ConfigMap to be used to
-                  back a configmap-server registry. Only used when SourceType = SourceTypeConfigmap
-                  or SourceTypeInternal.
-                type: string
-              description:
-                type: string
-              displayName:
-                description: Metadata
-                type: string
-              icon:
-                type: object
-                required:
-                - base64data
-                - mediatype
-                properties:
-                  base64data:
-                    type: string
-                  mediatype:
-                    type: string
-              image:
-                description: Image is an operator-registry container image to instantiate
-                  a registry-server with. Only used when SourceType = SourceTypeGrpc.
-                  If present, the address field is ignored.
-                type: string
-              priority:
-                description: 'Priority field assigns a weight to the catalog source
-                  to prioritize them so that it can be consumed by the dependency
-                  resolver. Usage: Higher weight indicates that this catalog source
-                  is preferred over lower weighted catalog sources during dependency
-                  resolution. The range of the priority value can go from positive
-                  to negative in the range of int32. The default value to a catalog
-                  source with unassigned priority would be 0. The catalog source with
-                  the same priority values will be ranked lexicographically based
-                  on its name.'
-                type: integer
-              publisher:
-                type: string
-              secrets:
-                description: Secrets represent set of secrets that can be used to
-                  access the contents of the catalog. It is best to keep this list
-                  small, since each will need to be tried for every catalog entry.
-                type: array
-                items:
-                  type: string
-              sourceType:
-                description: SourceType is the type of source
-                type: string
-              updateStrategy:
-                description: UpdateStrategy defines how updated catalog source images
-                  can be discovered Consists of an interval that defines polling duration
-                  and an embedded strategy type
-                type: object
-                properties:
-                  registryPoll:
-                    type: object
-                    properties:
-                      interval:
-                        description: Interval is used to determine the time interval
-                          between checks of the latest catalog source version. The
-                          catalog operator polls to see if a new version of the catalog
-                          source is available. If available, the latest image is pulled
-                          and gRPC traffic is directed to the latest catalog source.
-                        type: string
-          status:
-            type: object
-            properties:
-              configMapReference:
-                type: object
-                required:
-                - name
-                - namespace
-                properties:
-                  lastUpdateTime:
-                    type: string
-                    format: date-time
-                  name:
-                    type: string
-                  namespace:
-                    type: string
-                  resourceVersion:
-                    type: string
-                  uid:
-                    description: UID is a type that holds unique ID values, including
-                      UUIDs.  Because we don't ONLY use UUIDs, this is an alias to
-                      string.  Being a type captures intent and helps make sure that
-                      UIDs and names do not get conflated.
-                    type: string
-              connectionState:
-                type: object
-                required:
-                - lastObservedState
-                properties:
-                  address:
-                    type: string
-                  lastConnect:
-                    type: string
-                    format: date-time
-                  lastObservedState:
-                    type: string
-              latestImageRegistryPoll:
-                description: The last time the CatalogSource image registry has been
-                  polled to ensure the image is up-to-date
-                type: string
-                format: date-time
-              message:
-                description: A human readable message indicating details about why
-                  the CatalogSource is in this condition.
-                type: string
-              reason:
-                description: Reason is the reason the CatalogSource was transitioned
-                  to its current state.
-                type: string
-              registryService:
-                type: object
-                properties:
-                  createdAt:
-                    type: string
-                    format: date-time
-                  port:
-                    type: string
-                  protocol:
-                    type: string
-                  serviceName:
-                    type: string
-                  serviceNamespace:
-                    type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: clusterserviceversions.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: ClusterServiceVersion
-    listKind: ClusterServiceVersionList
-    plural: clusterserviceversions
-    shortNames:
-    - csv
-    - csvs
-    singular: clusterserviceversion
-  scope: Namespaced
-  versions:
-  - additionalPrinterColumns:
-    - description: The name of the CSV
-      jsonPath: .spec.displayName
-      name: Display
-      type: string
-    - description: The version of the CSV
-      jsonPath: .spec.version
-      name: Version
-      type: string
-    - description: The name of a CSV that this one replaces
-      jsonPath: .spec.replaces
-      name: Replaces
-      type: string
-    - jsonPath: .status.phase
-      name: Phase
-      type: string
-    name: v1alpha1
-    schema:
-      openAPIV3Schema:
-        description: ClusterServiceVersion is a Custom Resource of type `ClusterServiceVersionSpec`.
-        type: object
-        required:
-        - metadata
-        - spec
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: ClusterServiceVersionSpec declarations tell OLM how to install
-              an operator that can manage apps for a given version.
-            type: object
-            required:
-            - displayName
-            - install
-            properties:
-              annotations:
-                description: Annotations is an unstructured key value map stored with
-                  a resource that may be set by external tools to store and retrieve
-                  arbitrary metadata.
-                type: object
-                additionalProperties:
-                  type: string
-              apiservicedefinitions:
-                description: APIServiceDefinitions declares all of the extension apis
-                  managed or required by an operator being ran by ClusterServiceVersion.
-                type: object
-                properties:
-                  owned:
-                    type: array
-                    items:
-                      description: APIServiceDescription provides details to OLM about
-                        apis provided via aggregation
-                      type: object
-                      required:
-                      - group
-                      - kind
-                      - name
-                      - version
-                      properties:
-                        actionDescriptors:
-                          type: array
-                          items:
-                            description: ActionDescriptor describes a declarative
-                              action that can be performed on a custom resource instance
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        containerPort:
-                          type: integer
-                          format: int32
-                        deploymentName:
-                          type: string
-                        description:
-                          type: string
-                        displayName:
-                          type: string
-                        group:
-                          type: string
-                        kind:
-                          type: string
-                        name:
-                          type: string
-                        resources:
-                          type: array
-                          items:
-                            description: APIResourceReference is a Kubernetes resource
-                              type used by a custom resource
-                            type: object
-                            required:
-                            - kind
-                            - name
-                            - version
-                            properties:
-                              kind:
-                                type: string
-                              name:
-                                type: string
-                              version:
-                                type: string
-                        specDescriptors:
-                          type: array
-                          items:
-                            description: SpecDescriptor describes a field in a spec
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        statusDescriptors:
-                          type: array
-                          items:
-                            description: StatusDescriptor describes a field in a status
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        version:
-                          type: string
-                  required:
-                    type: array
-                    items:
-                      description: APIServiceDescription provides details to OLM about
-                        apis provided via aggregation
-                      type: object
-                      required:
-                      - group
-                      - kind
-                      - name
-                      - version
-                      properties:
-                        actionDescriptors:
-                          type: array
-                          items:
-                            description: ActionDescriptor describes a declarative
-                              action that can be performed on a custom resource instance
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        containerPort:
-                          type: integer
-                          format: int32
-                        deploymentName:
-                          type: string
-                        description:
-                          type: string
-                        displayName:
-                          type: string
-                        group:
-                          type: string
-                        kind:
-                          type: string
-                        name:
-                          type: string
-                        resources:
-                          type: array
-                          items:
-                            description: APIResourceReference is a Kubernetes resource
-                              type used by a custom resource
-                            type: object
-                            required:
-                            - kind
-                            - name
-                            - version
-                            properties:
-                              kind:
-                                type: string
-                              name:
-                                type: string
-                              version:
-                                type: string
-                        specDescriptors:
-                          type: array
-                          items:
-                            description: SpecDescriptor describes a field in a spec
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        statusDescriptors:
-                          type: array
-                          items:
-                            description: StatusDescriptor describes a field in a status
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        version:
-                          type: string
-              customresourcedefinitions:
-                description: "CustomResourceDefinitions declares all of the CRDs managed
-                  or required by an operator being ran by ClusterServiceVersion. \n
-                  If the CRD is present in the Owned list, it is implicitly required."
-                type: object
-                properties:
-                  owned:
-                    type: array
-                    items:
-                      description: CRDDescription provides details to OLM about the
-                        CRDs
-                      type: object
-                      required:
-                      - kind
-                      - name
-                      - version
-                      properties:
-                        actionDescriptors:
-                          type: array
-                          items:
-                            description: ActionDescriptor describes a declarative
-                              action that can be performed on a custom resource instance
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        description:
-                          type: string
-                        displayName:
-                          type: string
-                        kind:
-                          type: string
-                        name:
-                          type: string
-                        resources:
-                          type: array
-                          items:
-                            description: APIResourceReference is a Kubernetes resource
-                              type used by a custom resource
-                            type: object
-                            required:
-                            - kind
-                            - name
-                            - version
-                            properties:
-                              kind:
-                                type: string
-                              name:
-                                type: string
-                              version:
-                                type: string
-                        specDescriptors:
-                          type: array
-                          items:
-                            description: SpecDescriptor describes a field in a spec
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        statusDescriptors:
-                          type: array
-                          items:
-                            description: StatusDescriptor describes a field in a status
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        version:
-                          type: string
-                  required:
-                    type: array
-                    items:
-                      description: CRDDescription provides details to OLM about the
-                        CRDs
-                      type: object
-                      required:
-                      - kind
-                      - name
-                      - version
-                      properties:
-                        actionDescriptors:
-                          type: array
-                          items:
-                            description: ActionDescriptor describes a declarative
-                              action that can be performed on a custom resource instance
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        description:
-                          type: string
-                        displayName:
-                          type: string
-                        kind:
-                          type: string
-                        name:
-                          type: string
-                        resources:
-                          type: array
-                          items:
-                            description: APIResourceReference is a Kubernetes resource
-                              type used by a custom resource
-                            type: object
-                            required:
-                            - kind
-                            - name
-                            - version
-                            properties:
-                              kind:
-                                type: string
-                              name:
-                                type: string
-                              version:
-                                type: string
-                        specDescriptors:
-                          type: array
-                          items:
-                            description: SpecDescriptor describes a field in a spec
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        statusDescriptors:
-                          type: array
-                          items:
-                            description: StatusDescriptor describes a field in a status
-                              block of a CRD so that OLM can consume it
-                            type: object
-                            required:
-                            - path
-                            properties:
-                              description:
-                                type: string
-                              displayName:
-                                type: string
-                              path:
-                                type: string
-                              value:
-                                description: RawMessage is a raw encoded JSON value.
-                                  It implements Marshaler and Unmarshaler and can
-                                  be used to delay JSON decoding or precompute a JSON
-                                  encoding.
-                                type: string
-                                format: byte
-                              x-descriptors:
-                                type: array
-                                items:
-                                  type: string
-                        version:
-                          type: string
-              description:
-                type: string
-              displayName:
-                type: string
-              icon:
-                type: array
-                items:
-                  type: object
-                  required:
-                  - base64data
-                  - mediatype
-                  properties:
-                    base64data:
-                      type: string
-                    mediatype:
-                      type: string
-              install:
-                description: NamedInstallStrategy represents the block of an ClusterServiceVersion
-                  resource where the install strategy is specified.
-                type: object
-                required:
-                - strategy
-                properties:
-                  spec:
-                    description: StrategyDetailsDeployment represents the parsed details
-                      of a Deployment InstallStrategy.
-                    type: object
-                    required:
-                    - deployments
-                    properties:
-                      clusterPermissions:
-                        type: array
-                        items:
-                          description: StrategyDeploymentPermissions describe the
-                            rbac rules and service account needed by the install strategy
-                          type: object
-                          required:
-                          - rules
-                          - serviceAccountName
-                          properties:
-                            rules:
-                              type: array
-                              items:
-                                description: PolicyRule holds information that describes
-                                  a policy rule, but does not contain information
-                                  about who the rule applies to or which namespace
-                                  the rule applies to.
-                                type: object
-                                required:
-                                - verbs
-                                properties:
-                                  apiGroups:
-                                    description: APIGroups is the name of the APIGroup
-                                      that contains the resources.  If multiple API
-                                      groups are specified, any action requested against
-                                      one of the enumerated resources in any API group
-                                      will be allowed.
-                                    type: array
-                                    items:
-                                      type: string
-                                  nonResourceURLs:
-                                    description: NonResourceURLs is a set of partial
-                                      urls that a user should have access to.  *s
-                                      are allowed, but only as the full, final step
-                                      in the path Since non-resource URLs are not
-                                      namespaced, this field is only applicable for
-                                      ClusterRoles referenced from a ClusterRoleBinding.
-                                      Rules can either apply to API resources (such
-                                      as "pods" or "secrets") or non-resource URL
-                                      paths (such as "/api"),  but not both.
-                                    type: array
-                                    items:
-                                      type: string
-                                  resourceNames:
-                                    description: ResourceNames is an optional white
-                                      list of names that the rule applies to.  An
-                                      empty set means that everything is allowed.
-                                    type: array
-                                    items:
-                                      type: string
-                                  resources:
-                                    description: Resources is a list of resources
-                                      this rule applies to.  ResourceAll represents
-                                      all resources.
-                                    type: array
-                                    items:
-                                      type: string
-                                  verbs:
-                                    description: Verbs is a list of Verbs that apply
-                                      to ALL the ResourceKinds and AttributeRestrictions
-                                      contained in this rule.  VerbAll represents
-                                      all kinds.
-                                    type: array
-                                    items:
-                                      type: string
-                            serviceAccountName:
-                              type: string
-                      deployments:
-                        type: array
-                        items:
-                          description: StrategyDeploymentSpec contains the name, spec
-                            and labels for the deployment ALM should create
-                          type: object
-                          required:
-                          - name
-                          - spec
-                          properties:
-                            label:
-                              description: Set is a map of label:value. It implements
-                                Labels.
-                              type: object
-                              additionalProperties:
-                                type: string
-                            name:
-                              type: string
-                            spec:
-                              description: DeploymentSpec is the specification of
-                                the desired behavior of the Deployment.
-                              type: object
-                              required:
-                              - selector
-                              - template
-                              properties:
-                                minReadySeconds:
-                                  description: Minimum number of seconds for which
-                                    a newly created pod should be ready without any
-                                    of its container crashing, for it to be considered
-                                    available. Defaults to 0 (pod will be considered
-                                    available as soon as it is ready)
-                                  type: integer
-                                  format: int32
-                                paused:
-                                  description: Indicates that the deployment is paused.
-                                  type: boolean
-                                progressDeadlineSeconds:
-                                  description: The maximum time in seconds for a deployment
-                                    to make progress before it is considered to be
-                                    failed. The deployment controller will continue
-                                    to process failed deployments and a condition
-                                    with a ProgressDeadlineExceeded reason will be
-                                    surfaced in the deployment status. Note that progress
-                                    will not be estimated during the time a deployment
-                                    is paused. Defaults to 600s.
-                                  type: integer
-                                  format: int32
-                                replicas:
-                                  description: Number of desired pods. This is a pointer
-                                    to distinguish between explicit zero and not specified.
-                                    Defaults to 1.
-                                  type: integer
-                                  format: int32
-                                revisionHistoryLimit:
-                                  description: The number of old ReplicaSets to retain
-                                    to allow rollback. This is a pointer to distinguish
-                                    between explicit zero and not specified. Defaults
-                                    to 10.
-                                  type: integer
-                                  format: int32
-                                selector:
-                                  description: Label selector for pods. Existing ReplicaSets
-                                    whose pods are selected by this will be the ones
-                                    affected by this deployment. It must match the
-                                    pod template's labels.
-                                  type: object
-                                  properties:
-                                    matchExpressions:
-                                      description: matchExpressions is a list of label
-                                        selector requirements. The requirements are
-                                        ANDed.
-                                      type: array
-                                      items:
-                                        description: A label selector requirement
-                                          is a selector that contains values, a key,
-                                          and an operator that relates the key and
-                                          values.
-                                        type: object
-                                        required:
-                                        - key
-                                        - operator
-                                        properties:
-                                          key:
-                                            description: key is the label key that
-                                              the selector applies to.
-                                            type: string
-                                          operator:
-                                            description: operator represents a key's
-                                              relationship to a set of values. Valid
-                                              operators are In, NotIn, Exists and
-                                              DoesNotExist.
-                                            type: string
-                                          values:
-                                            description: values is an array of string
-                                              values. If the operator is In or NotIn,
-                                              the values array must be non-empty.
-                                              If the operator is Exists or DoesNotExist,
-                                              the values array must be empty. This
-                                              array is replaced during a strategic
-                                              merge patch.
-                                            type: array
-                                            items:
-                                              type: string
-                                    matchLabels:
-                                      description: matchLabels is a map of {key,value}
-                                        pairs. A single {key,value} in the matchLabels
-                                        map is equivalent to an element of matchExpressions,
-                                        whose key field is "key", the operator is
-                                        "In", and the values array contains only "value".
-                                        The requirements are ANDed.
-                                      type: object
-                                      additionalProperties:
-                                        type: string
-                                strategy:
-                                  description: The deployment strategy to use to replace
-                                    existing pods with new ones.
-                                  type: object
-                                  properties:
-                                    rollingUpdate:
-                                      description: 'Rolling update config params.
-                                        Present only if DeploymentStrategyType = RollingUpdate.
-                                        --- TODO: Update this to follow our convention
-                                        for oneOf, whatever we decide it to be.'
-                                      type: object
-                                      properties:
-                                        maxSurge:
-                                          description: 'The maximum number of pods
-                                            that can be scheduled above the desired
-                                            number of pods. Value can be an absolute
-                                            number (ex: 5) or a percentage of desired
-                                            pods (ex: 10%). This can not be 0 if MaxUnavailable
-                                            is 0. Absolute number is calculated from
-                                            percentage by rounding up. Defaults to
-                                            25%. Example: when this is set to 30%,
-                                            the new ReplicaSet can be scaled up immediately
-                                            when the rolling update starts, such that
-                                            the total number of old and new pods do
-                                            not exceed 130% of desired pods. Once
-                                            old pods have been killed, new ReplicaSet
-                                            can be scaled up further, ensuring that
-                                            total number of pods running at any time
-                                            during the update is at most 130% of desired
-                                            pods.'
-                                          anyOf:
-                                          - type: integer
-                                          - type: string
-                                          x-kubernetes-int-or-string: true
-                                        maxUnavailable:
-                                          description: 'The maximum number of pods
-                                            that can be unavailable during the update.
-                                            Value can be an absolute number (ex: 5)
-                                            or a percentage of desired pods (ex: 10%).
-                                            Absolute number is calculated from percentage
-                                            by rounding down. This can not be 0 if
-                                            MaxSurge is 0. Defaults to 25%. Example:
-                                            when this is set to 30%, the old ReplicaSet
-                                            can be scaled down to 70% of desired pods
-                                            immediately when the rolling update starts.
-                                            Once new pods are ready, old ReplicaSet
-                                            can be scaled down further, followed by
-                                            scaling up the new ReplicaSet, ensuring
-                                            that the total number of pods available
-                                            at all times during the update is at least
-                                            70% of desired pods.'
-                                          anyOf:
-                                          - type: integer
-                                          - type: string
-                                          x-kubernetes-int-or-string: true
-                                    type:
-                                      description: Type of deployment. Can be "Recreate"
-                                        or "RollingUpdate". Default is RollingUpdate.
-                                      type: string
-                                template:
-                                  description: Template describes the pods that will
-                                    be created.
-                                  type: object
-                                  properties:
-                                    metadata:
-                                      description: 'Standard object''s metadata. More
-                                        info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata'
-                                      type: object
-                                      x-kubernetes-preserve-unknown-fields: true
-                                    spec:
-                                      description: 'Specification of the desired behavior
-                                        of the pod. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status'
-                                      type: object
-                                      required:
-                                      - containers
-                                      properties:
-                                        activeDeadlineSeconds:
-                                          description: Optional duration in seconds
-                                            the pod may be active on the node relative
-                                            to StartTime before the system will actively
-                                            try to mark it failed and kill associated
-                                            containers. Value must be a positive integer.
-                                          type: integer
-                                          format: int64
-                                        affinity:
-                                          description: If specified, the pod's scheduling
-                                            constraints
-                                          type: object
-                                          properties:
-                                            nodeAffinity:
-                                              description: Describes node affinity
-                                                scheduling rules for the pod.
-                                              type: object
-                                              properties:
-                                                preferredDuringSchedulingIgnoredDuringExecution:
-                                                  description: The scheduler will
-                                                    prefer to schedule pods to nodes
-                                                    that satisfy the affinity expressions
-                                                    specified by this field, but it
-                                                    may choose a node that violates
-                                                    one or more of the expressions.
-                                                    The node that is most preferred
-                                                    is the one with the greatest sum
-                                                    of weights, i.e. for each node
-                                                    that meets all of the scheduling
-                                                    requirements (resource request,
-                                                    requiredDuringScheduling affinity
-                                                    expressions, etc.), compute a
-                                                    sum by iterating through the elements
-                                                    of this field and adding "weight"
-                                                    to the sum if the node matches
-                                                    the corresponding matchExpressions;
-                                                    the node(s) with the highest sum
-                                                    are the most preferred.
-                                                  type: array
-                                                  items:
-                                                    description: An empty preferred
-                                                      scheduling term matches all
-                                                      objects with implicit weight
-                                                      0 (i.e. it's a no-op). A null
-                                                      preferred scheduling term matches
-                                                      no objects (i.e. is also a no-op).
-                                                    type: object
-                                                    required:
-                                                    - preference
-                                                    - weight
-                                                    properties:
-                                                      preference:
-                                                        description: A node selector
-                                                          term, associated with the
-                                                          corresponding weight.
-                                                        type: object
-                                                        properties:
-                                                          matchExpressions:
-                                                            description: A list of
-                                                              node selector requirements
-                                                              by node's labels.
-                                                            type: array
-                                                            items:
-                                                              description: A node
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: The
-                                                                    label key that
-                                                                    the selector applies
-                                                                    to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: Represents
-                                                                    a key's relationship
-                                                                    to a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists, DoesNotExist.
-                                                                    Gt, and Lt.
-                                                                  type: string
-                                                                values:
-                                                                  description: An
-                                                                    array of string
-                                                                    values. If the
-                                                                    operator is In
-                                                                    or NotIn, the
-                                                                    values array must
-                                                                    be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    If the operator
-                                                                    is Gt or Lt, the
-                                                                    values array must
-                                                                    have a single
-                                                                    element, which
-                                                                    will be interpreted
-                                                                    as an integer.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                                          matchFields:
-                                                            description: A list of
-                                                              node selector requirements
-                                                              by node's fields.
-                                                            type: array
-                                                            items:
-                                                              description: A node
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: The
-                                                                    label key that
-                                                                    the selector applies
-                                                                    to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: Represents
-                                                                    a key's relationship
-                                                                    to a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists, DoesNotExist.
-                                                                    Gt, and Lt.
-                                                                  type: string
-                                                                values:
-                                                                  description: An
-                                                                    array of string
-                                                                    values. If the
-                                                                    operator is In
-                                                                    or NotIn, the
-                                                                    values array must
-                                                                    be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    If the operator
-                                                                    is Gt or Lt, the
-                                                                    values array must
-                                                                    have a single
-                                                                    element, which
-                                                                    will be interpreted
-                                                                    as an integer.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                                      weight:
-                                                        description: Weight associated
-                                                          with matching the corresponding
-                                                          nodeSelectorTerm, in the
-                                                          range 1-100.
-                                                        type: integer
-                                                        format: int32
-                                                requiredDuringSchedulingIgnoredDuringExecution:
-                                                  description: If the affinity requirements
-                                                    specified by this field are not
-                                                    met at scheduling time, the pod
-                                                    will not be scheduled onto the
-                                                    node. If the affinity requirements
-                                                    specified by this field cease
-                                                    to be met at some point during
-                                                    pod execution (e.g. due to an
-                                                    update), the system may or may
-                                                    not try to eventually evict the
-                                                    pod from its node.
-                                                  type: object
-                                                  required:
-                                                  - nodeSelectorTerms
-                                                  properties:
-                                                    nodeSelectorTerms:
-                                                      description: Required. A list
-                                                        of node selector terms. The
-                                                        terms are ORed.
-                                                      type: array
-                                                      items:
-                                                        description: A null or empty
-                                                          node selector term matches
-                                                          no objects. The requirements
-                                                          of them are ANDed. The TopologySelectorTerm
-                                                          type implements a subset
-                                                          of the NodeSelectorTerm.
-                                                        type: object
-                                                        properties:
-                                                          matchExpressions:
-                                                            description: A list of
-                                                              node selector requirements
-                                                              by node's labels.
-                                                            type: array
-                                                            items:
-                                                              description: A node
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: The
-                                                                    label key that
-                                                                    the selector applies
-                                                                    to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: Represents
-                                                                    a key's relationship
-                                                                    to a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists, DoesNotExist.
-                                                                    Gt, and Lt.
-                                                                  type: string
-                                                                values:
-                                                                  description: An
-                                                                    array of string
-                                                                    values. If the
-                                                                    operator is In
-                                                                    or NotIn, the
-                                                                    values array must
-                                                                    be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    If the operator
-                                                                    is Gt or Lt, the
-                                                                    values array must
-                                                                    have a single
-                                                                    element, which
-                                                                    will be interpreted
-                                                                    as an integer.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                                          matchFields:
-                                                            description: A list of
-                                                              node selector requirements
-                                                              by node's fields.
-                                                            type: array
-                                                            items:
-                                                              description: A node
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: The
-                                                                    label key that
-                                                                    the selector applies
-                                                                    to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: Represents
-                                                                    a key's relationship
-                                                                    to a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists, DoesNotExist.
-                                                                    Gt, and Lt.
-                                                                  type: string
-                                                                values:
-                                                                  description: An
-                                                                    array of string
-                                                                    values. If the
-                                                                    operator is In
-                                                                    or NotIn, the
-                                                                    values array must
-                                                                    be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    If the operator
-                                                                    is Gt or Lt, the
-                                                                    values array must
-                                                                    have a single
-                                                                    element, which
-                                                                    will be interpreted
-                                                                    as an integer.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                            podAffinity:
-                                              description: Describes pod affinity
-                                                scheduling rules (e.g. co-locate this
-                                                pod in the same node, zone, etc. as
-                                                some other pod(s)).
-                                              type: object
-                                              properties:
-                                                preferredDuringSchedulingIgnoredDuringExecution:
-                                                  description: The scheduler will
-                                                    prefer to schedule pods to nodes
-                                                    that satisfy the affinity expressions
-                                                    specified by this field, but it
-                                                    may choose a node that violates
-                                                    one or more of the expressions.
-                                                    The node that is most preferred
-                                                    is the one with the greatest sum
-                                                    of weights, i.e. for each node
-                                                    that meets all of the scheduling
-                                                    requirements (resource request,
-                                                    requiredDuringScheduling affinity
-                                                    expressions, etc.), compute a
-                                                    sum by iterating through the elements
-                                                    of this field and adding "weight"
-                                                    to the sum if the node has pods
-                                                    which matches the corresponding
-                                                    podAffinityTerm; the node(s) with
-                                                    the highest sum are the most preferred.
-                                                  type: array
-                                                  items:
-                                                    description: The weights of all
-                                                      of the matched WeightedPodAffinityTerm
-                                                      fields are added per-node to
-                                                      find the most preferred node(s)
-                                                    type: object
-                                                    required:
-                                                    - podAffinityTerm
-                                                    - weight
-                                                    properties:
-                                                      podAffinityTerm:
-                                                        description: Required. A pod
-                                                          affinity term, associated
-                                                          with the corresponding weight.
-                                                        type: object
-                                                        required:
-                                                        - topologyKey
-                                                        properties:
-                                                          labelSelector:
-                                                            description: A label query
-                                                              over a set of resources,
-                                                              in this case pods.
-                                                            type: object
-                                                            properties:
-                                                              matchExpressions:
-                                                                description: matchExpressions
-                                                                  is a list of label
-                                                                  selector requirements.
-                                                                  The requirements
-                                                                  are ANDed.
-                                                                type: array
-                                                                items:
-                                                                  description: A label
-                                                                    selector requirement
-                                                                    is a selector
-                                                                    that contains
-                                                                    values, a key,
-                                                                    and an operator
-                                                                    that relates the
-                                                                    key and values.
-                                                                  type: object
-                                                                  required:
-                                                                  - key
-                                                                  - operator
-                                                                  properties:
-                                                                    key:
-                                                                      description: key
-                                                                        is the label
-                                                                        key that the
-                                                                        selector applies
-                                                                        to.
-                                                                      type: string
-                                                                    operator:
-                                                                      description: operator
-                                                                        represents
-                                                                        a key's relationship
-                                                                        to a set of
-                                                                        values. Valid
-                                                                        operators
-                                                                        are In, NotIn,
-                                                                        Exists and
-                                                                        DoesNotExist.
-                                                                      type: string
-                                                                    values:
-                                                                      description: values
-                                                                        is an array
-                                                                        of string
-                                                                        values. If
-                                                                        the operator
-                                                                        is In or NotIn,
-                                                                        the values
-                                                                        array must
-                                                                        be non-empty.
-                                                                        If the operator
-                                                                        is Exists
-                                                                        or DoesNotExist,
-                                                                        the values
-                                                                        array must
-                                                                        be empty.
-                                                                        This array
-                                                                        is replaced
-                                                                        during a strategic
-                                                                        merge patch.
-                                                                      type: array
-                                                                      items:
-                                                                        type: string
-                                                              matchLabels:
-                                                                description: matchLabels
-                                                                  is a map of {key,value}
-                                                                  pairs. A single
-                                                                  {key,value} in the
-                                                                  matchLabels map
-                                                                  is equivalent to
-                                                                  an element of matchExpressions,
-                                                                  whose key field
-                                                                  is "key", the operator
-                                                                  is "In", and the
-                                                                  values array contains
-                                                                  only "value". The
-                                                                  requirements are
-                                                                  ANDed.
-                                                                type: object
-                                                                additionalProperties:
-                                                                  type: string
-                                                          namespaces:
-                                                            description: namespaces
-                                                              specifies which namespaces
-                                                              the labelSelector applies
-                                                              to (matches against);
-                                                              null or empty list means
-                                                              "this pod's namespace"
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                          topologyKey:
-                                                            description: This pod
-                                                              should be co-located
-                                                              (affinity) or not co-located
-                                                              (anti-affinity) with
-                                                              the pods matching the
-                                                              labelSelector in the
-                                                              specified namespaces,
-                                                              where co-located is
-                                                              defined as running on
-                                                              a node whose value of
-                                                              the label with key topologyKey
-                                                              matches that of any
-                                                              node on which any of
-                                                              the selected pods is
-                                                              running. Empty topologyKey
-                                                              is not allowed.
-                                                            type: string
-                                                      weight:
-                                                        description: weight associated
-                                                          with matching the corresponding
-                                                          podAffinityTerm, in the
-                                                          range 1-100.
-                                                        type: integer
-                                                        format: int32
-                                                requiredDuringSchedulingIgnoredDuringExecution:
-                                                  description: If the affinity requirements
-                                                    specified by this field are not
-                                                    met at scheduling time, the pod
-                                                    will not be scheduled onto the
-                                                    node. If the affinity requirements
-                                                    specified by this field cease
-                                                    to be met at some point during
-                                                    pod execution (e.g. due to a pod
-                                                    label update), the system may
-                                                    or may not try to eventually evict
-                                                    the pod from its node. When there
-                                                    are multiple elements, the lists
-                                                    of nodes corresponding to each
-                                                    podAffinityTerm are intersected,
-                                                    i.e. all terms must be satisfied.
-                                                  type: array
-                                                  items:
-                                                    description: Defines a set of
-                                                      pods (namely those matching
-                                                      the labelSelector relative to
-                                                      the given namespace(s)) that
-                                                      this pod should be co-located
-                                                      (affinity) or not co-located
-                                                      (anti-affinity) with, where
-                                                      co-located is defined as running
-                                                      on a node whose value of the
-                                                      label with key <topologyKey>
-                                                      matches that of any node on
-                                                      which a pod of the set of pods
-                                                      is running
-                                                    type: object
-                                                    required:
-                                                    - topologyKey
-                                                    properties:
-                                                      labelSelector:
-                                                        description: A label query
-                                                          over a set of resources,
-                                                          in this case pods.
-                                                        type: object
-                                                        properties:
-                                                          matchExpressions:
-                                                            description: matchExpressions
-                                                              is a list of label selector
-                                                              requirements. The requirements
-                                                              are ANDed.
-                                                            type: array
-                                                            items:
-                                                              description: A label
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: key
-                                                                    is the label key
-                                                                    that the selector
-                                                                    applies to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: operator
-                                                                    represents a key's
-                                                                    relationship to
-                                                                    a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists and DoesNotExist.
-                                                                  type: string
-                                                                values:
-                                                                  description: values
-                                                                    is an array of
-                                                                    string values.
-                                                                    If the operator
-                                                                    is In or NotIn,
-                                                                    the values array
-                                                                    must be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                                          matchLabels:
-                                                            description: matchLabels
-                                                              is a map of {key,value}
-                                                              pairs. A single {key,value}
-                                                              in the matchLabels map
-                                                              is equivalent to an
-                                                              element of matchExpressions,
-                                                              whose key field is "key",
-                                                              the operator is "In",
-                                                              and the values array
-                                                              contains only "value".
-                                                              The requirements are
-                                                              ANDed.
-                                                            type: object
-                                                            additionalProperties:
-                                                              type: string
-                                                      namespaces:
-                                                        description: namespaces specifies
-                                                          which namespaces the labelSelector
-                                                          applies to (matches against);
-                                                          null or empty list means
-                                                          "this pod's namespace"
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                      topologyKey:
-                                                        description: This pod should
-                                                          be co-located (affinity)
-                                                          or not co-located (anti-affinity)
-                                                          with the pods matching the
-                                                          labelSelector in the specified
-                                                          namespaces, where co-located
-                                                          is defined as running on
-                                                          a node whose value of the
-                                                          label with key topologyKey
-                                                          matches that of any node
-                                                          on which any of the selected
-                                                          pods is running. Empty topologyKey
-                                                          is not allowed.
-                                                        type: string
-                                            podAntiAffinity:
-                                              description: Describes pod anti-affinity
-                                                scheduling rules (e.g. avoid putting
-                                                this pod in the same node, zone, etc.
-                                                as some other pod(s)).
-                                              type: object
-                                              properties:
-                                                preferredDuringSchedulingIgnoredDuringExecution:
-                                                  description: The scheduler will
-                                                    prefer to schedule pods to nodes
-                                                    that satisfy the anti-affinity
-                                                    expressions specified by this
-                                                    field, but it may choose a node
-                                                    that violates one or more of the
-                                                    expressions. The node that is
-                                                    most preferred is the one with
-                                                    the greatest sum of weights, i.e.
-                                                    for each node that meets all of
-                                                    the scheduling requirements (resource
-                                                    request, requiredDuringScheduling
-                                                    anti-affinity expressions, etc.),
-                                                    compute a sum by iterating through
-                                                    the elements of this field and
-                                                    adding "weight" to the sum if
-                                                    the node has pods which matches
-                                                    the corresponding podAffinityTerm;
-                                                    the node(s) with the highest sum
-                                                    are the most preferred.
-                                                  type: array
-                                                  items:
-                                                    description: The weights of all
-                                                      of the matched WeightedPodAffinityTerm
-                                                      fields are added per-node to
-                                                      find the most preferred node(s)
-                                                    type: object
-                                                    required:
-                                                    - podAffinityTerm
-                                                    - weight
-                                                    properties:
-                                                      podAffinityTerm:
-                                                        description: Required. A pod
-                                                          affinity term, associated
-                                                          with the corresponding weight.
-                                                        type: object
-                                                        required:
-                                                        - topologyKey
-                                                        properties:
-                                                          labelSelector:
-                                                            description: A label query
-                                                              over a set of resources,
-                                                              in this case pods.
-                                                            type: object
-                                                            properties:
-                                                              matchExpressions:
-                                                                description: matchExpressions
-                                                                  is a list of label
-                                                                  selector requirements.
-                                                                  The requirements
-                                                                  are ANDed.
-                                                                type: array
-                                                                items:
-                                                                  description: A label
-                                                                    selector requirement
-                                                                    is a selector
-                                                                    that contains
-                                                                    values, a key,
-                                                                    and an operator
-                                                                    that relates the
-                                                                    key and values.
-                                                                  type: object
-                                                                  required:
-                                                                  - key
-                                                                  - operator
-                                                                  properties:
-                                                                    key:
-                                                                      description: key
-                                                                        is the label
-                                                                        key that the
-                                                                        selector applies
-                                                                        to.
-                                                                      type: string
-                                                                    operator:
-                                                                      description: operator
-                                                                        represents
-                                                                        a key's relationship
-                                                                        to a set of
-                                                                        values. Valid
-                                                                        operators
-                                                                        are In, NotIn,
-                                                                        Exists and
-                                                                        DoesNotExist.
-                                                                      type: string
-                                                                    values:
-                                                                      description: values
-                                                                        is an array
-                                                                        of string
-                                                                        values. If
-                                                                        the operator
-                                                                        is In or NotIn,
-                                                                        the values
-                                                                        array must
-                                                                        be non-empty.
-                                                                        If the operator
-                                                                        is Exists
-                                                                        or DoesNotExist,
-                                                                        the values
-                                                                        array must
-                                                                        be empty.
-                                                                        This array
-                                                                        is replaced
-                                                                        during a strategic
-                                                                        merge patch.
-                                                                      type: array
-                                                                      items:
-                                                                        type: string
-                                                              matchLabels:
-                                                                description: matchLabels
-                                                                  is a map of {key,value}
-                                                                  pairs. A single
-                                                                  {key,value} in the
-                                                                  matchLabels map
-                                                                  is equivalent to
-                                                                  an element of matchExpressions,
-                                                                  whose key field
-                                                                  is "key", the operator
-                                                                  is "In", and the
-                                                                  values array contains
-                                                                  only "value". The
-                                                                  requirements are
-                                                                  ANDed.
-                                                                type: object
-                                                                additionalProperties:
-                                                                  type: string
-                                                          namespaces:
-                                                            description: namespaces
-                                                              specifies which namespaces
-                                                              the labelSelector applies
-                                                              to (matches against);
-                                                              null or empty list means
-                                                              "this pod's namespace"
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                          topologyKey:
-                                                            description: This pod
-                                                              should be co-located
-                                                              (affinity) or not co-located
-                                                              (anti-affinity) with
-                                                              the pods matching the
-                                                              labelSelector in the
-                                                              specified namespaces,
-                                                              where co-located is
-                                                              defined as running on
-                                                              a node whose value of
-                                                              the label with key topologyKey
-                                                              matches that of any
-                                                              node on which any of
-                                                              the selected pods is
-                                                              running. Empty topologyKey
-                                                              is not allowed.
-                                                            type: string
-                                                      weight:
-                                                        description: weight associated
-                                                          with matching the corresponding
-                                                          podAffinityTerm, in the
-                                                          range 1-100.
-                                                        type: integer
-                                                        format: int32
-                                                requiredDuringSchedulingIgnoredDuringExecution:
-                                                  description: If the anti-affinity
-                                                    requirements specified by this
-                                                    field are not met at scheduling
-                                                    time, the pod will not be scheduled
-                                                    onto the node. If the anti-affinity
-                                                    requirements specified by this
-                                                    field cease to be met at some
-                                                    point during pod execution (e.g.
-                                                    due to a pod label update), the
-                                                    system may or may not try to eventually
-                                                    evict the pod from its node. When
-                                                    there are multiple elements, the
-                                                    lists of nodes corresponding to
-                                                    each podAffinityTerm are intersected,
-                                                    i.e. all terms must be satisfied.
-                                                  type: array
-                                                  items:
-                                                    description: Defines a set of
-                                                      pods (namely those matching
-                                                      the labelSelector relative to
-                                                      the given namespace(s)) that
-                                                      this pod should be co-located
-                                                      (affinity) or not co-located
-                                                      (anti-affinity) with, where
-                                                      co-located is defined as running
-                                                      on a node whose value of the
-                                                      label with key <topologyKey>
-                                                      matches that of any node on
-                                                      which a pod of the set of pods
-                                                      is running
-                                                    type: object
-                                                    required:
-                                                    - topologyKey
-                                                    properties:
-                                                      labelSelector:
-                                                        description: A label query
-                                                          over a set of resources,
-                                                          in this case pods.
-                                                        type: object
-                                                        properties:
-                                                          matchExpressions:
-                                                            description: matchExpressions
-                                                              is a list of label selector
-                                                              requirements. The requirements
-                                                              are ANDed.
-                                                            type: array
-                                                            items:
-                                                              description: A label
-                                                                selector requirement
-                                                                is a selector that
-                                                                contains values, a
-                                                                key, and an operator
-                                                                that relates the key
-                                                                and values.
-                                                              type: object
-                                                              required:
-                                                              - key
-                                                              - operator
-                                                              properties:
-                                                                key:
-                                                                  description: key
-                                                                    is the label key
-                                                                    that the selector
-                                                                    applies to.
-                                                                  type: string
-                                                                operator:
-                                                                  description: operator
-                                                                    represents a key's
-                                                                    relationship to
-                                                                    a set of values.
-                                                                    Valid operators
-                                                                    are In, NotIn,
-                                                                    Exists and DoesNotExist.
-                                                                  type: string
-                                                                values:
-                                                                  description: values
-                                                                    is an array of
-                                                                    string values.
-                                                                    If the operator
-                                                                    is In or NotIn,
-                                                                    the values array
-                                                                    must be non-empty.
-                                                                    If the operator
-                                                                    is Exists or DoesNotExist,
-                                                                    the values array
-                                                                    must be empty.
-                                                                    This array is
-                                                                    replaced during
-                                                                    a strategic merge
-                                                                    patch.
-                                                                  type: array
-                                                                  items:
-                                                                    type: string
-                                                          matchLabels:
-                                                            description: matchLabels
-                                                              is a map of {key,value}
-                                                              pairs. A single {key,value}
-                                                              in the matchLabels map
-                                                              is equivalent to an
-                                                              element of matchExpressions,
-                                                              whose key field is "key",
-                                                              the operator is "In",
-                                                              and the values array
-                                                              contains only "value".
-                                                              The requirements are
-                                                              ANDed.
-                                                            type: object
-                                                            additionalProperties:
-                                                              type: string
-                                                      namespaces:
-                                                        description: namespaces specifies
-                                                          which namespaces the labelSelector
-                                                          applies to (matches against);
-                                                          null or empty list means
-                                                          "this pod's namespace"
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                      topologyKey:
-                                                        description: This pod should
-                                                          be co-located (affinity)
-                                                          or not co-located (anti-affinity)
-                                                          with the pods matching the
-                                                          labelSelector in the specified
-                                                          namespaces, where co-located
-                                                          is defined as running on
-                                                          a node whose value of the
-                                                          label with key topologyKey
-                                                          matches that of any node
-                                                          on which any of the selected
-                                                          pods is running. Empty topologyKey
-                                                          is not allowed.
-                                                        type: string
-                                        automountServiceAccountToken:
-                                          description: AutomountServiceAccountToken
-                                            indicates whether a service account token
-                                            should be automatically mounted.
-                                          type: boolean
-                                        containers:
-                                          description: List of containers belonging
-                                            to the pod. Containers cannot currently
-                                            be added or removed. There must be at
-                                            least one container in a Pod. Cannot be
-                                            updated.
-                                          type: array
-                                          items:
-                                            description: A single application container
-                                              that you want to run within a pod.
-                                            type: object
-                                            required:
-                                            - name
-                                            properties:
-                                              args:
-                                                description: 'Arguments to the entrypoint.
-                                                  The docker image''s CMD is used
-                                                  if this is not provided. Variable
-                                                  references $(VAR_NAME) are expanded
-                                                  using the container''s environment.
-                                                  If a variable cannot be resolved,
-                                                  the reference in the input string
-                                                  will be unchanged. The $(VAR_NAME)
-                                                  syntax can be escaped with a double
-                                                  $$, ie: $$(VAR_NAME). Escaped references
-                                                  will never be expanded, regardless
-                                                  of whether the variable exists or
-                                                  not. Cannot be updated. More info:
-                                                  https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              command:
-                                                description: 'Entrypoint array. Not
-                                                  executed within a shell. The docker
-                                                  image''s ENTRYPOINT is used if this
-                                                  is not provided. Variable references
-                                                  $(VAR_NAME) are expanded using the
-                                                  container''s environment. If a variable
-                                                  cannot be resolved, the reference
-                                                  in the input string will be unchanged.
-                                                  The $(VAR_NAME) syntax can be escaped
-                                                  with a double $$, ie: $$(VAR_NAME).
-                                                  Escaped references will never be
-                                                  expanded, regardless of whether
-                                                  the variable exists or not. Cannot
-                                                  be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              env:
-                                                description: List of environment variables
-                                                  to set in the container. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvVar represents an
-                                                    environment variable present in
-                                                    a Container.
-                                                  type: object
-                                                  required:
-                                                  - name
-                                                  properties:
-                                                    name:
-                                                      description: Name of the environment
-                                                        variable. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    value:
-                                                      description: 'Variable references
-                                                        $(VAR_NAME) are expanded using
-                                                        the previous defined environment
-                                                        variables in the container
-                                                        and any service environment
-                                                        variables. If a variable cannot
-                                                        be resolved, the reference
-                                                        in the input string will be
-                                                        unchanged. The $(VAR_NAME)
-                                                        syntax can be escaped with
-                                                        a double $$, ie: $$(VAR_NAME).
-                                                        Escaped references will never
-                                                        be expanded, regardless of
-                                                        whether the variable exists
-                                                        or not. Defaults to "".'
-                                                      type: string
-                                                    valueFrom:
-                                                      description: Source for the
-                                                        environment variable's value.
-                                                        Cannot be used if value is
-                                                        not empty.
-                                                      type: object
-                                                      properties:
-                                                        configMapKeyRef:
-                                                          description: Selects a key
-                                                            of a ConfigMap.
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                to select.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the ConfigMap
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                                        fieldRef:
-                                                          description: 'Selects a
-                                                            field of the pod: supports
-                                                            metadata.name, metadata.namespace,
-                                                            metadata.labels, metadata.annotations,
-                                                            spec.nodeName, spec.serviceAccountName,
-                                                            status.hostIP, status.podIP,
-                                                            status.podIPs.'
-                                                          type: object
-                                                          required:
-                                                          - fieldPath
-                                                          properties:
-                                                            apiVersion:
-                                                              description: Version
-                                                                of the schema the
-                                                                FieldPath is written
-                                                                in terms of, defaults
-                                                                to "v1".
-                                                              type: string
-                                                            fieldPath:
-                                                              description: Path of
-                                                                the field to select
-                                                                in the specified API
-                                                                version.
-                                                              type: string
-                                                        resourceFieldRef:
-                                                          description: 'Selects a
-                                                            resource of the container:
-                                                            only resources limits
-                                                            and requests (limits.cpu,
-                                                            limits.memory, limits.ephemeral-storage,
-                                                            requests.cpu, requests.memory
-                                                            and requests.ephemeral-storage)
-                                                            are currently supported.'
-                                                          type: object
-                                                          required:
-                                                          - resource
-                                                          properties:
-                                                            containerName:
-                                                              description: 'Container
-                                                                name: required for
-                                                                volumes, optional
-                                                                for env vars'
-                                                              type: string
-                                                            divisor:
-                                                              description: Specifies
-                                                                the output format
-                                                                of the exposed resources,
-                                                                defaults to "1"
-                                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                              anyOf:
-                                                              - type: integer
-                                                              - type: string
-                                                              x-kubernetes-int-or-string: true
-                                                            resource:
-                                                              description: 'Required:
-                                                                resource to select'
-                                                              type: string
-                                                        secretKeyRef:
-                                                          description: Selects a key
-                                                            of a secret in the pod's
-                                                            namespace
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                of the secret to select
-                                                                from.  Must be a valid
-                                                                secret key.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the Secret
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                              envFrom:
-                                                description: List of sources to populate
-                                                  environment variables in the container.
-                                                  The keys defined within a source
-                                                  must be a C_IDENTIFIER. All invalid
-                                                  keys will be reported as an event
-                                                  when the container is starting.
-                                                  When a key exists in multiple sources,
-                                                  the value associated with the last
-                                                  source will take precedence. Values
-                                                  defined by an Env with a duplicate
-                                                  key will take precedence. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvFromSource represents
-                                                    the source of a set of ConfigMaps
-                                                  type: object
-                                                  properties:
-                                                    configMapRef:
-                                                      description: The ConfigMap to
-                                                        select from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the ConfigMap must be
-                                                            defined
-                                                          type: boolean
-                                                    prefix:
-                                                      description: An optional identifier
-                                                        to prepend to each key in
-                                                        the ConfigMap. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    secretRef:
-                                                      description: The Secret to select
-                                                        from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the Secret must be defined
-                                                          type: boolean
-                                              image:
-                                                description: 'Docker image name. More
-                                                  info: https://kubernetes.io/docs/concepts/containers/images
-                                                  This field is optional to allow
-                                                  higher level config management to
-                                                  default or override container images
-                                                  in workload controllers like Deployments
-                                                  and StatefulSets.'
-                                                type: string
-                                              imagePullPolicy:
-                                                description: 'Image pull policy. One
-                                                  of Always, Never, IfNotPresent.
-                                                  Defaults to Always if :latest tag
-                                                  is specified, or IfNotPresent otherwise.
-                                                  Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
-                                                type: string
-                                              lifecycle:
-                                                description: Actions that the management
-                                                  system should take in response to
-                                                  container lifecycle events. Cannot
-                                                  be updated.
-                                                type: object
-                                                properties:
-                                                  postStart:
-                                                    description: 'PostStart is called
-                                                      immediately after a container
-                                                      is created. If the handler fails,
-                                                      the container is terminated
-                                                      and restarted according to its
-                                                      restart policy. Other management
-                                                      of the container blocks until
-                                                      the hook completes. More info:
-                                                      https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                  preStop:
-                                                    description: 'PreStop is called
-                                                      immediately before a container
-                                                      is terminated due to an API
-                                                      request or management event
-                                                      such as liveness/startup probe
-                                                      failure, preemption, resource
-                                                      contention, etc. The handler
-                                                      is not called if the container
-                                                      crashes or exits. The reason
-                                                      for termination is passed to
-                                                      the handler. The Pod''s termination
-                                                      grace period countdown begins
-                                                      before the PreStop hooked is
-                                                      executed. Regardless of the
-                                                      outcome of the handler, the
-                                                      container will eventually terminate
-                                                      within the Pod''s termination
-                                                      grace period. Other management
-                                                      of the container blocks until
-                                                      the hook completes or until
-                                                      the termination grace period
-                                                      is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                              livenessProbe:
-                                                description: 'Periodic probe of container
-                                                  liveness. Container will be restarted
-                                                  if the probe fails. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              name:
-                                                description: Name of the container
-                                                  specified as a DNS_LABEL. Each container
-                                                  in a pod must have a unique name
-                                                  (DNS_LABEL). Cannot be updated.
-                                                type: string
-                                              ports:
-                                                description: List of ports to expose
-                                                  from the container. Exposing a port
-                                                  here gives the system additional
-                                                  information about the network connections
-                                                  a container uses, but is primarily
-                                                  informational. Not specifying a
-                                                  port here DOES NOT prevent that
-                                                  port from being exposed. Any port
-                                                  which is listening on the default
-                                                  "0.0.0.0" address inside a container
-                                                  will be accessible from the network.
-                                                  Cannot be updated.
-                                                type: array
-                                                items:
-                                                  description: ContainerPort represents
-                                                    a network port in a single container.
-                                                  type: object
-                                                  required:
-                                                  - containerPort
-                                                  properties:
-                                                    containerPort:
-                                                      description: Number of port
-                                                        to expose on the pod's IP
-                                                        address. This must be a valid
-                                                        port number, 0 < x < 65536.
-                                                      type: integer
-                                                      format: int32
-                                                    hostIP:
-                                                      description: What host IP to
-                                                        bind the external port to.
-                                                      type: string
-                                                    hostPort:
-                                                      description: Number of port
-                                                        to expose on the host. If
-                                                        specified, this must be a
-                                                        valid port number, 0 < x <
-                                                        65536. If HostNetwork is specified,
-                                                        this must match ContainerPort.
-                                                        Most containers do not need
-                                                        this.
-                                                      type: integer
-                                                      format: int32
-                                                    name:
-                                                      description: If specified, this
-                                                        must be an IANA_SVC_NAME and
-                                                        unique within the pod. Each
-                                                        named port in a pod must have
-                                                        a unique name. Name for the
-                                                        port that can be referred
-                                                        to by services.
-                                                      type: string
-                                                    protocol:
-                                                      description: Protocol for port.
-                                                        Must be UDP, TCP, or SCTP.
-                                                        Defaults to "TCP".
-                                                      type: string
-                                                      default: TCP
-                                                x-kubernetes-list-map-keys:
-                                                - containerPort
-                                                - protocol
-                                                x-kubernetes-list-type: map
-                                              readinessProbe:
-                                                description: 'Periodic probe of container
-                                                  service readiness. Container will
-                                                  be removed from service endpoints
-                                                  if the probe fails. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              resources:
-                                                description: 'Compute Resources required
-                                                  by this container. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                type: object
-                                                properties:
-                                                  limits:
-                                                    description: 'Limits describes
-                                                      the maximum amount of compute
-                                                      resources allowed. More info:
-                                                      https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                                  requests:
-                                                    description: 'Requests describes
-                                                      the minimum amount of compute
-                                                      resources required. If Requests
-                                                      is omitted for a container,
-                                                      it defaults to Limits if that
-                                                      is explicitly specified, otherwise
-                                                      to an implementation-defined
-                                                      value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                              securityContext:
-                                                description: 'Security options the
-                                                  pod should run with. More info:
-                                                  https://kubernetes.io/docs/concepts/policy/security-context/
-                                                  More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/'
-                                                type: object
-                                                properties:
-                                                  allowPrivilegeEscalation:
-                                                    description: 'AllowPrivilegeEscalation
-                                                      controls whether a process can
-                                                      gain more privileges than its
-                                                      parent process. This bool directly
-                                                      controls if the no_new_privs
-                                                      flag will be set on the container
-                                                      process. AllowPrivilegeEscalation
-                                                      is true always when the container
-                                                      is: 1) run as Privileged 2)
-                                                      has CAP_SYS_ADMIN'
-                                                    type: boolean
-                                                  capabilities:
-                                                    description: The capabilities
-                                                      to add/drop when running containers.
-                                                      Defaults to the default set
-                                                      of capabilities granted by the
-                                                      container runtime.
-                                                    type: object
-                                                    properties:
-                                                      add:
-                                                        description: Added capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                      drop:
-                                                        description: Removed capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                  privileged:
-                                                    description: Run container in
-                                                      privileged mode. Processes in
-                                                      privileged containers are essentially
-                                                      equivalent to root on the host.
-                                                      Defaults to false.
-                                                    type: boolean
-                                                  procMount:
-                                                    description: procMount denotes
-                                                      the type of proc mount to use
-                                                      for the containers. The default
-                                                      is DefaultProcMount which uses
-                                                      the container runtime defaults
-                                                      for readonly paths and masked
-                                                      paths. This requires the ProcMountType
-                                                      feature flag to be enabled.
-                                                    type: string
-                                                  readOnlyRootFilesystem:
-                                                    description: Whether this container
-                                                      has a read-only root filesystem.
-                                                      Default is false.
-                                                    type: boolean
-                                                  runAsGroup:
-                                                    description: The GID to run the
-                                                      entrypoint of the container
-                                                      process. Uses runtime default
-                                                      if unset. May also be set in
-                                                      PodSecurityContext.  If set
-                                                      in both SecurityContext and
-                                                      PodSecurityContext, the value
-                                                      specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  runAsNonRoot:
-                                                    description: Indicates that the
-                                                      container must run as a non-root
-                                                      user. If true, the Kubelet will
-                                                      validate the image at runtime
-                                                      to ensure that it does not run
-                                                      as UID 0 (root) and fail to
-                                                      start the container if it does.
-                                                      If unset or false, no such validation
-                                                      will be performed. May also
-                                                      be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: boolean
-                                                  runAsUser:
-                                                    description: The UID to run the
-                                                      entrypoint of the container
-                                                      process. Defaults to user specified
-                                                      in image metadata if unspecified.
-                                                      May also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  seLinuxOptions:
-                                                    description: The SELinux context
-                                                      to be applied to the container.
-                                                      If unspecified, the container
-                                                      runtime will allocate a random
-                                                      SELinux context for each container.  May
-                                                      also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      level:
-                                                        description: Level is SELinux
-                                                          level label that applies
-                                                          to the container.
-                                                        type: string
-                                                      role:
-                                                        description: Role is a SELinux
-                                                          role label that applies
-                                                          to the container.
-                                                        type: string
-                                                      type:
-                                                        description: Type is a SELinux
-                                                          type label that applies
-                                                          to the container.
-                                                        type: string
-                                                      user:
-                                                        description: User is a SELinux
-                                                          user label that applies
-                                                          to the container.
-                                                        type: string
-                                                  windowsOptions:
-                                                    description: The Windows specific
-                                                      settings applied to all containers.
-                                                      If unspecified, the options
-                                                      from the PodSecurityContext
-                                                      will be used. If set in both
-                                                      SecurityContext and PodSecurityContext,
-                                                      the value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      gmsaCredentialSpec:
-                                                        description: GMSACredentialSpec
-                                                          is where the GMSA admission
-                                                          webhook (https://github.com/kubernetes-sigs/windows-gmsa)
-                                                          inlines the contents of
-                                                          the GMSA credential spec
-                                                          named by the GMSACredentialSpecName
-                                                          field.
-                                                        type: string
-                                                      gmsaCredentialSpecName:
-                                                        description: GMSACredentialSpecName
-                                                          is the name of the GMSA
-                                                          credential spec to use.
-                                                        type: string
-                                                      runAsUserName:
-                                                        description: The UserName
-                                                          in Windows to run the entrypoint
-                                                          of the container process.
-                                                          Defaults to the user specified
-                                                          in image metadata if unspecified.
-                                                          May also be set in PodSecurityContext.
-                                                          If set in both SecurityContext
-                                                          and PodSecurityContext,
-                                                          the value specified in SecurityContext
-                                                          takes precedence.
-                                                        type: string
-                                              startupProbe:
-                                                description: 'StartupProbe indicates
-                                                  that the Pod has successfully initialized.
-                                                  If specified, no other probes are
-                                                  executed until this completes successfully.
-                                                  If this probe fails, the Pod will
-                                                  be restarted, just as if the livenessProbe
-                                                  failed. This can be used to provide
-                                                  different probe parameters at the
-                                                  beginning of a Pod''s lifecycle,
-                                                  when it might take a long time to
-                                                  load data or warm a cache, than
-                                                  during steady-state operation. This
-                                                  cannot be updated. This is a beta
-                                                  feature enabled by the StartupProbe
-                                                  feature flag. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              stdin:
-                                                description: Whether this container
-                                                  should allocate a buffer for stdin
-                                                  in the container runtime. If this
-                                                  is not set, reads from stdin in
-                                                  the container will always result
-                                                  in EOF. Default is false.
-                                                type: boolean
-                                              stdinOnce:
-                                                description: Whether the container
-                                                  runtime should close the stdin channel
-                                                  after it has been opened by a single
-                                                  attach. When stdin is true the stdin
-                                                  stream will remain open across multiple
-                                                  attach sessions. If stdinOnce is
-                                                  set to true, stdin is opened on
-                                                  container start, is empty until
-                                                  the first client attaches to stdin,
-                                                  and then remains open and accepts
-                                                  data until the client disconnects,
-                                                  at which time stdin is closed and
-                                                  remains closed until the container
-                                                  is restarted. If this flag is false,
-                                                  a container processes that reads
-                                                  from stdin will never receive an
-                                                  EOF. Default is false
-                                                type: boolean
-                                              terminationMessagePath:
-                                                description: 'Optional: Path at which
-                                                  the file to which the container''s
-                                                  termination message will be written
-                                                  is mounted into the container''s
-                                                  filesystem. Message written is intended
-                                                  to be brief final status, such as
-                                                  an assertion failure message. Will
-                                                  be truncated by the node if greater
-                                                  than 4096 bytes. The total message
-                                                  length across all containers will
-                                                  be limited to 12kb. Defaults to
-                                                  /dev/termination-log. Cannot be
-                                                  updated.'
-                                                type: string
-                                              terminationMessagePolicy:
-                                                description: Indicate how the termination
-                                                  message should be populated. File
-                                                  will use the contents of terminationMessagePath
-                                                  to populate the container status
-                                                  message on both success and failure.
-                                                  FallbackToLogsOnError will use the
-                                                  last chunk of container log output
-                                                  if the termination message file
-                                                  is empty and the container exited
-                                                  with an error. The log output is
-                                                  limited to 2048 bytes or 80 lines,
-                                                  whichever is smaller. Defaults to
-                                                  File. Cannot be updated.
-                                                type: string
-                                              tty:
-                                                description: Whether this container
-                                                  should allocate a TTY for itself,
-                                                  also requires 'stdin' to be true.
-                                                  Default is false.
-                                                type: boolean
-                                              volumeDevices:
-                                                description: volumeDevices is the
-                                                  list of block devices to be used
-                                                  by the container.
-                                                type: array
-                                                items:
-                                                  description: volumeDevice describes
-                                                    a mapping of a raw block device
-                                                    within a container.
-                                                  type: object
-                                                  required:
-                                                  - devicePath
-                                                  - name
-                                                  properties:
-                                                    devicePath:
-                                                      description: devicePath is the
-                                                        path inside of the container
-                                                        that the device will be mapped
-                                                        to.
-                                                      type: string
-                                                    name:
-                                                      description: name must match
-                                                        the name of a persistentVolumeClaim
-                                                        in the pod
-                                                      type: string
-                                              volumeMounts:
-                                                description: Pod volumes to mount
-                                                  into the container's filesystem.
-                                                  Cannot be updated.
-                                                type: array
-                                                items:
-                                                  description: VolumeMount describes
-                                                    a mounting of a Volume within
-                                                    a container.
-                                                  type: object
-                                                  required:
-                                                  - mountPath
-                                                  - name
-                                                  properties:
-                                                    mountPath:
-                                                      description: Path within the
-                                                        container at which the volume
-                                                        should be mounted.  Must not
-                                                        contain ':'.
-                                                      type: string
-                                                    mountPropagation:
-                                                      description: mountPropagation
-                                                        determines how mounts are
-                                                        propagated from the host to
-                                                        container and the other way
-                                                        around. When not set, MountPropagationNone
-                                                        is used. This field is beta
-                                                        in 1.10.
-                                                      type: string
-                                                    name:
-                                                      description: This must match
-                                                        the Name of a Volume.
-                                                      type: string
-                                                    readOnly:
-                                                      description: Mounted read-only
-                                                        if true, read-write otherwise
-                                                        (false or unspecified). Defaults
-                                                        to false.
-                                                      type: boolean
-                                                    subPath:
-                                                      description: Path within the
-                                                        volume from which the container's
-                                                        volume should be mounted.
-                                                        Defaults to "" (volume's root).
-                                                      type: string
-                                                    subPathExpr:
-                                                      description: Expanded path within
-                                                        the volume from which the
-                                                        container's volume should
-                                                        be mounted. Behaves similarly
-                                                        to SubPath but environment
-                                                        variable references $(VAR_NAME)
-                                                        are expanded using the container's
-                                                        environment. Defaults to ""
-                                                        (volume's root). SubPathExpr
-                                                        and SubPath are mutually exclusive.
-                                                      type: string
-                                              workingDir:
-                                                description: Container's working directory.
-                                                  If not specified, the container
-                                                  runtime's default will be used,
-                                                  which might be configured in the
-                                                  container image. Cannot be updated.
-                                                type: string
-                                        dnsConfig:
-                                          description: Specifies the DNS parameters
-                                            of a pod. Parameters specified here will
-                                            be merged to the generated DNS configuration
-                                            based on DNSPolicy.
-                                          type: object
-                                          properties:
-                                            nameservers:
-                                              description: A list of DNS name server
-                                                IP addresses. This will be appended
-                                                to the base nameservers generated
-                                                from DNSPolicy. Duplicated nameservers
-                                                will be removed.
-                                              type: array
-                                              items:
-                                                type: string
-                                            options:
-                                              description: A list of DNS resolver
-                                                options. This will be merged with
-                                                the base options generated from DNSPolicy.
-                                                Duplicated entries will be removed.
-                                                Resolution options given in Options
-                                                will override those that appear in
-                                                the base DNSPolicy.
-                                              type: array
-                                              items:
-                                                description: PodDNSConfigOption defines
-                                                  DNS resolver options of a pod.
-                                                type: object
-                                                properties:
-                                                  name:
-                                                    description: Required.
-                                                    type: string
-                                                  value:
-                                                    type: string
-                                            searches:
-                                              description: A list of DNS search domains
-                                                for host-name lookup. This will be
-                                                appended to the base search paths
-                                                generated from DNSPolicy. Duplicated
-                                                search paths will be removed.
-                                              type: array
-                                              items:
-                                                type: string
-                                        dnsPolicy:
-                                          description: Set DNS policy for the pod.
-                                            Defaults to "ClusterFirst". Valid values
-                                            are 'ClusterFirstWithHostNet', 'ClusterFirst',
-                                            'Default' or 'None'. DNS parameters given
-                                            in DNSConfig will be merged with the policy
-                                            selected with DNSPolicy. To have DNS options
-                                            set along with hostNetwork, you have to
-                                            specify DNS policy explicitly to 'ClusterFirstWithHostNet'.
-                                          type: string
-                                        enableServiceLinks:
-                                          description: 'EnableServiceLinks indicates
-                                            whether information about services should
-                                            be injected into pod''s environment variables,
-                                            matching the syntax of Docker links. Optional:
-                                            Defaults to true.'
-                                          type: boolean
-                                        ephemeralContainers:
-                                          description: List of ephemeral containers
-                                            run in this pod. Ephemeral containers
-                                            may be run in an existing pod to perform
-                                            user-initiated actions such as debugging.
-                                            This list cannot be specified when creating
-                                            a pod, and it cannot be modified by updating
-                                            the pod spec. In order to add an ephemeral
-                                            container to an existing pod, use the
-                                            pod's ephemeralcontainers subresource.
-                                            This field is alpha-level and is only
-                                            honored by servers that enable the EphemeralContainers
-                                            feature.
-                                          type: array
-                                          items:
-                                            description: An EphemeralContainer is
-                                              a container that may be added temporarily
-                                              to an existing pod for user-initiated
-                                              activities such as debugging. Ephemeral
-                                              containers have no resource or scheduling
-                                              guarantees, and they will not be restarted
-                                              when they exit or when a pod is removed
-                                              or restarted. If an ephemeral container
-                                              causes a pod to exceed its resource
-                                              allocation, the pod may be evicted.
-                                              Ephemeral containers may not be added
-                                              by directly updating the pod spec. They
-                                              must be added via the pod's ephemeralcontainers
-                                              subresource, and they will appear in
-                                              the pod spec once added. This is an
-                                              alpha feature enabled by the EphemeralContainers
-                                              feature flag.
-                                            type: object
-                                            required:
-                                            - name
-                                            properties:
-                                              args:
-                                                description: 'Arguments to the entrypoint.
-                                                  The docker image''s CMD is used
-                                                  if this is not provided. Variable
-                                                  references $(VAR_NAME) are expanded
-                                                  using the container''s environment.
-                                                  If a variable cannot be resolved,
-                                                  the reference in the input string
-                                                  will be unchanged. The $(VAR_NAME)
-                                                  syntax can be escaped with a double
-                                                  $$, ie: $$(VAR_NAME). Escaped references
-                                                  will never be expanded, regardless
-                                                  of whether the variable exists or
-                                                  not. Cannot be updated. More info:
-                                                  https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              command:
-                                                description: 'Entrypoint array. Not
-                                                  executed within a shell. The docker
-                                                  image''s ENTRYPOINT is used if this
-                                                  is not provided. Variable references
-                                                  $(VAR_NAME) are expanded using the
-                                                  container''s environment. If a variable
-                                                  cannot be resolved, the reference
-                                                  in the input string will be unchanged.
-                                                  The $(VAR_NAME) syntax can be escaped
-                                                  with a double $$, ie: $$(VAR_NAME).
-                                                  Escaped references will never be
-                                                  expanded, regardless of whether
-                                                  the variable exists or not. Cannot
-                                                  be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              env:
-                                                description: List of environment variables
-                                                  to set in the container. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvVar represents an
-                                                    environment variable present in
-                                                    a Container.
-                                                  type: object
-                                                  required:
-                                                  - name
-                                                  properties:
-                                                    name:
-                                                      description: Name of the environment
-                                                        variable. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    value:
-                                                      description: 'Variable references
-                                                        $(VAR_NAME) are expanded using
-                                                        the previous defined environment
-                                                        variables in the container
-                                                        and any service environment
-                                                        variables. If a variable cannot
-                                                        be resolved, the reference
-                                                        in the input string will be
-                                                        unchanged. The $(VAR_NAME)
-                                                        syntax can be escaped with
-                                                        a double $$, ie: $$(VAR_NAME).
-                                                        Escaped references will never
-                                                        be expanded, regardless of
-                                                        whether the variable exists
-                                                        or not. Defaults to "".'
-                                                      type: string
-                                                    valueFrom:
-                                                      description: Source for the
-                                                        environment variable's value.
-                                                        Cannot be used if value is
-                                                        not empty.
-                                                      type: object
-                                                      properties:
-                                                        configMapKeyRef:
-                                                          description: Selects a key
-                                                            of a ConfigMap.
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                to select.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the ConfigMap
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                                        fieldRef:
-                                                          description: 'Selects a
-                                                            field of the pod: supports
-                                                            metadata.name, metadata.namespace,
-                                                            metadata.labels, metadata.annotations,
-                                                            spec.nodeName, spec.serviceAccountName,
-                                                            status.hostIP, status.podIP,
-                                                            status.podIPs.'
-                                                          type: object
-                                                          required:
-                                                          - fieldPath
-                                                          properties:
-                                                            apiVersion:
-                                                              description: Version
-                                                                of the schema the
-                                                                FieldPath is written
-                                                                in terms of, defaults
-                                                                to "v1".
-                                                              type: string
-                                                            fieldPath:
-                                                              description: Path of
-                                                                the field to select
-                                                                in the specified API
-                                                                version.
-                                                              type: string
-                                                        resourceFieldRef:
-                                                          description: 'Selects a
-                                                            resource of the container:
-                                                            only resources limits
-                                                            and requests (limits.cpu,
-                                                            limits.memory, limits.ephemeral-storage,
-                                                            requests.cpu, requests.memory
-                                                            and requests.ephemeral-storage)
-                                                            are currently supported.'
-                                                          type: object
-                                                          required:
-                                                          - resource
-                                                          properties:
-                                                            containerName:
-                                                              description: 'Container
-                                                                name: required for
-                                                                volumes, optional
-                                                                for env vars'
-                                                              type: string
-                                                            divisor:
-                                                              description: Specifies
-                                                                the output format
-                                                                of the exposed resources,
-                                                                defaults to "1"
-                                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                              anyOf:
-                                                              - type: integer
-                                                              - type: string
-                                                              x-kubernetes-int-or-string: true
-                                                            resource:
-                                                              description: 'Required:
-                                                                resource to select'
-                                                              type: string
-                                                        secretKeyRef:
-                                                          description: Selects a key
-                                                            of a secret in the pod's
-                                                            namespace
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                of the secret to select
-                                                                from.  Must be a valid
-                                                                secret key.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the Secret
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                              envFrom:
-                                                description: List of sources to populate
-                                                  environment variables in the container.
-                                                  The keys defined within a source
-                                                  must be a C_IDENTIFIER. All invalid
-                                                  keys will be reported as an event
-                                                  when the container is starting.
-                                                  When a key exists in multiple sources,
-                                                  the value associated with the last
-                                                  source will take precedence. Values
-                                                  defined by an Env with a duplicate
-                                                  key will take precedence. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvFromSource represents
-                                                    the source of a set of ConfigMaps
-                                                  type: object
-                                                  properties:
-                                                    configMapRef:
-                                                      description: The ConfigMap to
-                                                        select from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the ConfigMap must be
-                                                            defined
-                                                          type: boolean
-                                                    prefix:
-                                                      description: An optional identifier
-                                                        to prepend to each key in
-                                                        the ConfigMap. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    secretRef:
-                                                      description: The Secret to select
-                                                        from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the Secret must be defined
-                                                          type: boolean
-                                              image:
-                                                description: 'Docker image name. More
-                                                  info: https://kubernetes.io/docs/concepts/containers/images'
-                                                type: string
-                                              imagePullPolicy:
-                                                description: 'Image pull policy. One
-                                                  of Always, Never, IfNotPresent.
-                                                  Defaults to Always if :latest tag
-                                                  is specified, or IfNotPresent otherwise.
-                                                  Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
-                                                type: string
-                                              lifecycle:
-                                                description: Lifecycle is not allowed
-                                                  for ephemeral containers.
-                                                type: object
-                                                properties:
-                                                  postStart:
-                                                    description: 'PostStart is called
-                                                      immediately after a container
-                                                      is created. If the handler fails,
-                                                      the container is terminated
-                                                      and restarted according to its
-                                                      restart policy. Other management
-                                                      of the container blocks until
-                                                      the hook completes. More info:
-                                                      https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                  preStop:
-                                                    description: 'PreStop is called
-                                                      immediately before a container
-                                                      is terminated due to an API
-                                                      request or management event
-                                                      such as liveness/startup probe
-                                                      failure, preemption, resource
-                                                      contention, etc. The handler
-                                                      is not called if the container
-                                                      crashes or exits. The reason
-                                                      for termination is passed to
-                                                      the handler. The Pod''s termination
-                                                      grace period countdown begins
-                                                      before the PreStop hooked is
-                                                      executed. Regardless of the
-                                                      outcome of the handler, the
-                                                      container will eventually terminate
-                                                      within the Pod''s termination
-                                                      grace period. Other management
-                                                      of the container blocks until
-                                                      the hook completes or until
-                                                      the termination grace period
-                                                      is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                              livenessProbe:
-                                                description: Probes are not allowed
-                                                  for ephemeral containers.
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              name:
-                                                description: Name of the ephemeral
-                                                  container specified as a DNS_LABEL.
-                                                  This name must be unique among all
-                                                  containers, init containers and
-                                                  ephemeral containers.
-                                                type: string
-                                              ports:
-                                                description: Ports are not allowed
-                                                  for ephemeral containers.
-                                                type: array
-                                                items:
-                                                  description: ContainerPort represents
-                                                    a network port in a single container.
-                                                  type: object
-                                                  required:
-                                                  - containerPort
-                                                  properties:
-                                                    containerPort:
-                                                      description: Number of port
-                                                        to expose on the pod's IP
-                                                        address. This must be a valid
-                                                        port number, 0 < x < 65536.
-                                                      type: integer
-                                                      format: int32
-                                                    hostIP:
-                                                      description: What host IP to
-                                                        bind the external port to.
-                                                      type: string
-                                                    hostPort:
-                                                      description: Number of port
-                                                        to expose on the host. If
-                                                        specified, this must be a
-                                                        valid port number, 0 < x <
-                                                        65536. If HostNetwork is specified,
-                                                        this must match ContainerPort.
-                                                        Most containers do not need
-                                                        this.
-                                                      type: integer
-                                                      format: int32
-                                                    name:
-                                                      description: If specified, this
-                                                        must be an IANA_SVC_NAME and
-                                                        unique within the pod. Each
-                                                        named port in a pod must have
-                                                        a unique name. Name for the
-                                                        port that can be referred
-                                                        to by services.
-                                                      type: string
-                                                    protocol:
-                                                      description: Protocol for port.
-                                                        Must be UDP, TCP, or SCTP.
-                                                        Defaults to "TCP".
-                                                      type: string
-                                              readinessProbe:
-                                                description: Probes are not allowed
-                                                  for ephemeral containers.
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              resources:
-                                                description: Resources are not allowed
-                                                  for ephemeral containers. Ephemeral
-                                                  containers use spare resources already
-                                                  allocated to the pod.
-                                                type: object
-                                                properties:
-                                                  limits:
-                                                    description: 'Limits describes
-                                                      the maximum amount of compute
-                                                      resources allowed. More info:
-                                                      https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                                  requests:
-                                                    description: 'Requests describes
-                                                      the minimum amount of compute
-                                                      resources required. If Requests
-                                                      is omitted for a container,
-                                                      it defaults to Limits if that
-                                                      is explicitly specified, otherwise
-                                                      to an implementation-defined
-                                                      value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                              securityContext:
-                                                description: SecurityContext is not
-                                                  allowed for ephemeral containers.
-                                                type: object
-                                                properties:
-                                                  allowPrivilegeEscalation:
-                                                    description: 'AllowPrivilegeEscalation
-                                                      controls whether a process can
-                                                      gain more privileges than its
-                                                      parent process. This bool directly
-                                                      controls if the no_new_privs
-                                                      flag will be set on the container
-                                                      process. AllowPrivilegeEscalation
-                                                      is true always when the container
-                                                      is: 1) run as Privileged 2)
-                                                      has CAP_SYS_ADMIN'
-                                                    type: boolean
-                                                  capabilities:
-                                                    description: The capabilities
-                                                      to add/drop when running containers.
-                                                      Defaults to the default set
-                                                      of capabilities granted by the
-                                                      container runtime.
-                                                    type: object
-                                                    properties:
-                                                      add:
-                                                        description: Added capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                      drop:
-                                                        description: Removed capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                  privileged:
-                                                    description: Run container in
-                                                      privileged mode. Processes in
-                                                      privileged containers are essentially
-                                                      equivalent to root on the host.
-                                                      Defaults to false.
-                                                    type: boolean
-                                                  procMount:
-                                                    description: procMount denotes
-                                                      the type of proc mount to use
-                                                      for the containers. The default
-                                                      is DefaultProcMount which uses
-                                                      the container runtime defaults
-                                                      for readonly paths and masked
-                                                      paths. This requires the ProcMountType
-                                                      feature flag to be enabled.
-                                                    type: string
-                                                  readOnlyRootFilesystem:
-                                                    description: Whether this container
-                                                      has a read-only root filesystem.
-                                                      Default is false.
-                                                    type: boolean
-                                                  runAsGroup:
-                                                    description: The GID to run the
-                                                      entrypoint of the container
-                                                      process. Uses runtime default
-                                                      if unset. May also be set in
-                                                      PodSecurityContext.  If set
-                                                      in both SecurityContext and
-                                                      PodSecurityContext, the value
-                                                      specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  runAsNonRoot:
-                                                    description: Indicates that the
-                                                      container must run as a non-root
-                                                      user. If true, the Kubelet will
-                                                      validate the image at runtime
-                                                      to ensure that it does not run
-                                                      as UID 0 (root) and fail to
-                                                      start the container if it does.
-                                                      If unset or false, no such validation
-                                                      will be performed. May also
-                                                      be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: boolean
-                                                  runAsUser:
-                                                    description: The UID to run the
-                                                      entrypoint of the container
-                                                      process. Defaults to user specified
-                                                      in image metadata if unspecified.
-                                                      May also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  seLinuxOptions:
-                                                    description: The SELinux context
-                                                      to be applied to the container.
-                                                      If unspecified, the container
-                                                      runtime will allocate a random
-                                                      SELinux context for each container.  May
-                                                      also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      level:
-                                                        description: Level is SELinux
-                                                          level label that applies
-                                                          to the container.
-                                                        type: string
-                                                      role:
-                                                        description: Role is a SELinux
-                                                          role label that applies
-                                                          to the container.
-                                                        type: string
-                                                      type:
-                                                        description: Type is a SELinux
-                                                          type label that applies
-                                                          to the container.
-                                                        type: string
-                                                      user:
-                                                        description: User is a SELinux
-                                                          user label that applies
-                                                          to the container.
-                                                        type: string
-                                                  windowsOptions:
-                                                    description: The Windows specific
-                                                      settings applied to all containers.
-                                                      If unspecified, the options
-                                                      from the PodSecurityContext
-                                                      will be used. If set in both
-                                                      SecurityContext and PodSecurityContext,
-                                                      the value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      gmsaCredentialSpec:
-                                                        description: GMSACredentialSpec
-                                                          is where the GMSA admission
-                                                          webhook (https://github.com/kubernetes-sigs/windows-gmsa)
-                                                          inlines the contents of
-                                                          the GMSA credential spec
-                                                          named by the GMSACredentialSpecName
-                                                          field.
-                                                        type: string
-                                                      gmsaCredentialSpecName:
-                                                        description: GMSACredentialSpecName
-                                                          is the name of the GMSA
-                                                          credential spec to use.
-                                                        type: string
-                                                      runAsUserName:
-                                                        description: The UserName
-                                                          in Windows to run the entrypoint
-                                                          of the container process.
-                                                          Defaults to the user specified
-                                                          in image metadata if unspecified.
-                                                          May also be set in PodSecurityContext.
-                                                          If set in both SecurityContext
-                                                          and PodSecurityContext,
-                                                          the value specified in SecurityContext
-                                                          takes precedence.
-                                                        type: string
-                                              startupProbe:
-                                                description: Probes are not allowed
-                                                  for ephemeral containers.
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              stdin:
-                                                description: Whether this container
-                                                  should allocate a buffer for stdin
-                                                  in the container runtime. If this
-                                                  is not set, reads from stdin in
-                                                  the container will always result
-                                                  in EOF. Default is false.
-                                                type: boolean
-                                              stdinOnce:
-                                                description: Whether the container
-                                                  runtime should close the stdin channel
-                                                  after it has been opened by a single
-                                                  attach. When stdin is true the stdin
-                                                  stream will remain open across multiple
-                                                  attach sessions. If stdinOnce is
-                                                  set to true, stdin is opened on
-                                                  container start, is empty until
-                                                  the first client attaches to stdin,
-                                                  and then remains open and accepts
-                                                  data until the client disconnects,
-                                                  at which time stdin is closed and
-                                                  remains closed until the container
-                                                  is restarted. If this flag is false,
-                                                  a container processes that reads
-                                                  from stdin will never receive an
-                                                  EOF. Default is false
-                                                type: boolean
-                                              targetContainerName:
-                                                description: If set, the name of the
-                                                  container from PodSpec that this
-                                                  ephemeral container targets. The
-                                                  ephemeral container will be run
-                                                  in the namespaces (IPC, PID, etc)
-                                                  of this container. If not set then
-                                                  the ephemeral container is run in
-                                                  whatever namespaces are shared for
-                                                  the pod. Note that the container
-                                                  runtime must support this feature.
-                                                type: string
-                                              terminationMessagePath:
-                                                description: 'Optional: Path at which
-                                                  the file to which the container''s
-                                                  termination message will be written
-                                                  is mounted into the container''s
-                                                  filesystem. Message written is intended
-                                                  to be brief final status, such as
-                                                  an assertion failure message. Will
-                                                  be truncated by the node if greater
-                                                  than 4096 bytes. The total message
-                                                  length across all containers will
-                                                  be limited to 12kb. Defaults to
-                                                  /dev/termination-log. Cannot be
-                                                  updated.'
-                                                type: string
-                                              terminationMessagePolicy:
-                                                description: Indicate how the termination
-                                                  message should be populated. File
-                                                  will use the contents of terminationMessagePath
-                                                  to populate the container status
-                                                  message on both success and failure.
-                                                  FallbackToLogsOnError will use the
-                                                  last chunk of container log output
-                                                  if the termination message file
-                                                  is empty and the container exited
-                                                  with an error. The log output is
-                                                  limited to 2048 bytes or 80 lines,
-                                                  whichever is smaller. Defaults to
-                                                  File. Cannot be updated.
-                                                type: string
-                                              tty:
-                                                description: Whether this container
-                                                  should allocate a TTY for itself,
-                                                  also requires 'stdin' to be true.
-                                                  Default is false.
-                                                type: boolean
-                                              volumeDevices:
-                                                description: volumeDevices is the
-                                                  list of block devices to be used
-                                                  by the container.
-                                                type: array
-                                                items:
-                                                  description: volumeDevice describes
-                                                    a mapping of a raw block device
-                                                    within a container.
-                                                  type: object
-                                                  required:
-                                                  - devicePath
-                                                  - name
-                                                  properties:
-                                                    devicePath:
-                                                      description: devicePath is the
-                                                        path inside of the container
-                                                        that the device will be mapped
-                                                        to.
-                                                      type: string
-                                                    name:
-                                                      description: name must match
-                                                        the name of a persistentVolumeClaim
-                                                        in the pod
-                                                      type: string
-                                              volumeMounts:
-                                                description: Pod volumes to mount
-                                                  into the container's filesystem.
-                                                  Cannot be updated.
-                                                type: array
-                                                items:
-                                                  description: VolumeMount describes
-                                                    a mounting of a Volume within
-                                                    a container.
-                                                  type: object
-                                                  required:
-                                                  - mountPath
-                                                  - name
-                                                  properties:
-                                                    mountPath:
-                                                      description: Path within the
-                                                        container at which the volume
-                                                        should be mounted.  Must not
-                                                        contain ':'.
-                                                      type: string
-                                                    mountPropagation:
-                                                      description: mountPropagation
-                                                        determines how mounts are
-                                                        propagated from the host to
-                                                        container and the other way
-                                                        around. When not set, MountPropagationNone
-                                                        is used. This field is beta
-                                                        in 1.10.
-                                                      type: string
-                                                    name:
-                                                      description: This must match
-                                                        the Name of a Volume.
-                                                      type: string
-                                                    readOnly:
-                                                      description: Mounted read-only
-                                                        if true, read-write otherwise
-                                                        (false or unspecified). Defaults
-                                                        to false.
-                                                      type: boolean
-                                                    subPath:
-                                                      description: Path within the
-                                                        volume from which the container's
-                                                        volume should be mounted.
-                                                        Defaults to "" (volume's root).
-                                                      type: string
-                                                    subPathExpr:
-                                                      description: Expanded path within
-                                                        the volume from which the
-                                                        container's volume should
-                                                        be mounted. Behaves similarly
-                                                        to SubPath but environment
-                                                        variable references $(VAR_NAME)
-                                                        are expanded using the container's
-                                                        environment. Defaults to ""
-                                                        (volume's root). SubPathExpr
-                                                        and SubPath are mutually exclusive.
-                                                      type: string
-                                              workingDir:
-                                                description: Container's working directory.
-                                                  If not specified, the container
-                                                  runtime's default will be used,
-                                                  which might be configured in the
-                                                  container image. Cannot be updated.
-                                                type: string
-                                        hostAliases:
-                                          description: HostAliases is an optional
-                                            list of hosts and IPs that will be injected
-                                            into the pod's hosts file if specified.
-                                            This is only valid for non-hostNetwork
-                                            pods.
-                                          type: array
-                                          items:
-                                            description: HostAlias holds the mapping
-                                              between IP and hostnames that will be
-                                              injected as an entry in the pod's hosts
-                                              file.
-                                            type: object
-                                            properties:
-                                              hostnames:
-                                                description: Hostnames for the above
-                                                  IP address.
-                                                type: array
-                                                items:
-                                                  type: string
-                                              ip:
-                                                description: IP address of the host
-                                                  file entry.
-                                                type: string
-                                        hostIPC:
-                                          description: 'Use the host''s ipc namespace.
-                                            Optional: Default to false.'
-                                          type: boolean
-                                        hostNetwork:
-                                          description: Host networking requested for
-                                            this pod. Use the host's network namespace.
-                                            If this option is set, the ports that
-                                            will be used must be specified. Default
-                                            to false.
-                                          type: boolean
-                                        hostPID:
-                                          description: 'Use the host''s pid namespace.
-                                            Optional: Default to false.'
-                                          type: boolean
-                                        hostname:
-                                          description: Specifies the hostname of the
-                                            Pod If not specified, the pod's hostname
-                                            will be set to a system-defined value.
-                                          type: string
-                                        imagePullSecrets:
-                                          description: 'ImagePullSecrets is an optional
-                                            list of references to secrets in the same
-                                            namespace to use for pulling any of the
-                                            images used by this PodSpec. If specified,
-                                            these secrets will be passed to individual
-                                            puller implementations for them to use.
-                                            For example, in the case of docker, only
-                                            DockerConfig type secrets are honored.
-                                            More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod'
-                                          type: array
-                                          items:
-                                            description: LocalObjectReference contains
-                                              enough information to let you locate
-                                              the referenced object inside the same
-                                              namespace.
-                                            type: object
-                                            properties:
-                                              name:
-                                                description: 'Name of the referent.
-                                                  More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                  TODO: Add other useful fields. apiVersion,
-                                                  kind, uid?'
-                                                type: string
-                                        initContainers:
-                                          description: 'List of initialization containers
-                                            belonging to the pod. Init containers
-                                            are executed in order prior to containers
-                                            being started. If any init container fails,
-                                            the pod is considered to have failed and
-                                            is handled according to its restartPolicy.
-                                            The name for an init container or normal
-                                            container must be unique among all containers.
-                                            Init containers may not have Lifecycle
-                                            actions, Readiness probes, Liveness probes,
-                                            or Startup probes. The resourceRequirements
-                                            of an init container are taken into account
-                                            during scheduling by finding the highest
-                                            request/limit for each resource type,
-                                            and then using the max of of that value
-                                            or the sum of the normal containers. Limits
-                                            are applied to init containers in a similar
-                                            fashion. Init containers cannot currently
-                                            be added or removed. Cannot be updated.
-                                            More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/'
-                                          type: array
-                                          items:
-                                            description: A single application container
-                                              that you want to run within a pod.
-                                            type: object
-                                            required:
-                                            - name
-                                            properties:
-                                              args:
-                                                description: 'Arguments to the entrypoint.
-                                                  The docker image''s CMD is used
-                                                  if this is not provided. Variable
-                                                  references $(VAR_NAME) are expanded
-                                                  using the container''s environment.
-                                                  If a variable cannot be resolved,
-                                                  the reference in the input string
-                                                  will be unchanged. The $(VAR_NAME)
-                                                  syntax can be escaped with a double
-                                                  $$, ie: $$(VAR_NAME). Escaped references
-                                                  will never be expanded, regardless
-                                                  of whether the variable exists or
-                                                  not. Cannot be updated. More info:
-                                                  https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              command:
-                                                description: 'Entrypoint array. Not
-                                                  executed within a shell. The docker
-                                                  image''s ENTRYPOINT is used if this
-                                                  is not provided. Variable references
-                                                  $(VAR_NAME) are expanded using the
-                                                  container''s environment. If a variable
-                                                  cannot be resolved, the reference
-                                                  in the input string will be unchanged.
-                                                  The $(VAR_NAME) syntax can be escaped
-                                                  with a double $$, ie: $$(VAR_NAME).
-                                                  Escaped references will never be
-                                                  expanded, regardless of whether
-                                                  the variable exists or not. Cannot
-                                                  be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
-                                                type: array
-                                                items:
-                                                  type: string
-                                              env:
-                                                description: List of environment variables
-                                                  to set in the container. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvVar represents an
-                                                    environment variable present in
-                                                    a Container.
-                                                  type: object
-                                                  required:
-                                                  - name
-                                                  properties:
-                                                    name:
-                                                      description: Name of the environment
-                                                        variable. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    value:
-                                                      description: 'Variable references
-                                                        $(VAR_NAME) are expanded using
-                                                        the previous defined environment
-                                                        variables in the container
-                                                        and any service environment
-                                                        variables. If a variable cannot
-                                                        be resolved, the reference
-                                                        in the input string will be
-                                                        unchanged. The $(VAR_NAME)
-                                                        syntax can be escaped with
-                                                        a double $$, ie: $$(VAR_NAME).
-                                                        Escaped references will never
-                                                        be expanded, regardless of
-                                                        whether the variable exists
-                                                        or not. Defaults to "".'
-                                                      type: string
-                                                    valueFrom:
-                                                      description: Source for the
-                                                        environment variable's value.
-                                                        Cannot be used if value is
-                                                        not empty.
-                                                      type: object
-                                                      properties:
-                                                        configMapKeyRef:
-                                                          description: Selects a key
-                                                            of a ConfigMap.
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                to select.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the ConfigMap
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                                        fieldRef:
-                                                          description: 'Selects a
-                                                            field of the pod: supports
-                                                            metadata.name, metadata.namespace,
-                                                            metadata.labels, metadata.annotations,
-                                                            spec.nodeName, spec.serviceAccountName,
-                                                            status.hostIP, status.podIP,
-                                                            status.podIPs.'
-                                                          type: object
-                                                          required:
-                                                          - fieldPath
-                                                          properties:
-                                                            apiVersion:
-                                                              description: Version
-                                                                of the schema the
-                                                                FieldPath is written
-                                                                in terms of, defaults
-                                                                to "v1".
-                                                              type: string
-                                                            fieldPath:
-                                                              description: Path of
-                                                                the field to select
-                                                                in the specified API
-                                                                version.
-                                                              type: string
-                                                        resourceFieldRef:
-                                                          description: 'Selects a
-                                                            resource of the container:
-                                                            only resources limits
-                                                            and requests (limits.cpu,
-                                                            limits.memory, limits.ephemeral-storage,
-                                                            requests.cpu, requests.memory
-                                                            and requests.ephemeral-storage)
-                                                            are currently supported.'
-                                                          type: object
-                                                          required:
-                                                          - resource
-                                                          properties:
-                                                            containerName:
-                                                              description: 'Container
-                                                                name: required for
-                                                                volumes, optional
-                                                                for env vars'
-                                                              type: string
-                                                            divisor:
-                                                              description: Specifies
-                                                                the output format
-                                                                of the exposed resources,
-                                                                defaults to "1"
-                                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                              anyOf:
-                                                              - type: integer
-                                                              - type: string
-                                                              x-kubernetes-int-or-string: true
-                                                            resource:
-                                                              description: 'Required:
-                                                                resource to select'
-                                                              type: string
-                                                        secretKeyRef:
-                                                          description: Selects a key
-                                                            of a secret in the pod's
-                                                            namespace
-                                                          type: object
-                                                          required:
-                                                          - key
-                                                          properties:
-                                                            key:
-                                                              description: The key
-                                                                of the secret to select
-                                                                from.  Must be a valid
-                                                                secret key.
-                                                              type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the Secret
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                              envFrom:
-                                                description: List of sources to populate
-                                                  environment variables in the container.
-                                                  The keys defined within a source
-                                                  must be a C_IDENTIFIER. All invalid
-                                                  keys will be reported as an event
-                                                  when the container is starting.
-                                                  When a key exists in multiple sources,
-                                                  the value associated with the last
-                                                  source will take precedence. Values
-                                                  defined by an Env with a duplicate
-                                                  key will take precedence. Cannot
-                                                  be updated.
-                                                type: array
-                                                items:
-                                                  description: EnvFromSource represents
-                                                    the source of a set of ConfigMaps
-                                                  type: object
-                                                  properties:
-                                                    configMapRef:
-                                                      description: The ConfigMap to
-                                                        select from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the ConfigMap must be
-                                                            defined
-                                                          type: boolean
-                                                    prefix:
-                                                      description: An optional identifier
-                                                        to prepend to each key in
-                                                        the ConfigMap. Must be a C_IDENTIFIER.
-                                                      type: string
-                                                    secretRef:
-                                                      description: The Secret to select
-                                                        from
-                                                      type: object
-                                                      properties:
-                                                        name:
-                                                          description: 'Name of the
-                                                            referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                            TODO: Add other useful
-                                                            fields. apiVersion, kind,
-                                                            uid?'
-                                                          type: string
-                                                        optional:
-                                                          description: Specify whether
-                                                            the Secret must be defined
-                                                          type: boolean
-                                              image:
-                                                description: 'Docker image name. More
-                                                  info: https://kubernetes.io/docs/concepts/containers/images
-                                                  This field is optional to allow
-                                                  higher level config management to
-                                                  default or override container images
-                                                  in workload controllers like Deployments
-                                                  and StatefulSets.'
-                                                type: string
-                                              imagePullPolicy:
-                                                description: 'Image pull policy. One
-                                                  of Always, Never, IfNotPresent.
-                                                  Defaults to Always if :latest tag
-                                                  is specified, or IfNotPresent otherwise.
-                                                  Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
-                                                type: string
-                                              lifecycle:
-                                                description: Actions that the management
-                                                  system should take in response to
-                                                  container lifecycle events. Cannot
-                                                  be updated.
-                                                type: object
-                                                properties:
-                                                  postStart:
-                                                    description: 'PostStart is called
-                                                      immediately after a container
-                                                      is created. If the handler fails,
-                                                      the container is terminated
-                                                      and restarted according to its
-                                                      restart policy. Other management
-                                                      of the container blocks until
-                                                      the hook completes. More info:
-                                                      https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                  preStop:
-                                                    description: 'PreStop is called
-                                                      immediately before a container
-                                                      is terminated due to an API
-                                                      request or management event
-                                                      such as liveness/startup probe
-                                                      failure, preemption, resource
-                                                      contention, etc. The handler
-                                                      is not called if the container
-                                                      crashes or exits. The reason
-                                                      for termination is passed to
-                                                      the handler. The Pod''s termination
-                                                      grace period countdown begins
-                                                      before the PreStop hooked is
-                                                      executed. Regardless of the
-                                                      outcome of the handler, the
-                                                      container will eventually terminate
-                                                      within the Pod''s termination
-                                                      grace period. Other management
-                                                      of the container blocks until
-                                                      the hook completes or until
-                                                      the termination grace period
-                                                      is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
-                                                    type: object
-                                                    properties:
-                                                      exec:
-                                                        description: One and only
-                                                          one of the following should
-                                                          be specified. Exec specifies
-                                                          the action to take.
-                                                        type: object
-                                                        properties:
-                                                          command:
-                                                            description: Command is
-                                                              the command line to
-                                                              execute inside the container,
-                                                              the working directory
-                                                              for the command  is
-                                                              root ('/') in the container's
-                                                              filesystem. The command
-                                                              is simply exec'd, it
-                                                              is not run inside a
-                                                              shell, so traditional
-                                                              shell instructions ('|',
-                                                              etc) won't work. To
-                                                              use a shell, you need
-                                                              to explicitly call out
-                                                              to that shell. Exit
-                                                              status of 0 is treated
-                                                              as live/healthy and
-                                                              non-zero is unhealthy.
-                                                            type: array
-                                                            items:
-                                                              type: string
-                                                      httpGet:
-                                                        description: HTTPGet specifies
-                                                          the http request to perform.
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: Host name
-                                                              to connect to, defaults
-                                                              to the pod IP. You probably
-                                                              want to set "Host" in
-                                                              httpHeaders instead.
-                                                            type: string
-                                                          httpHeaders:
-                                                            description: Custom headers
-                                                              to set in the request.
-                                                              HTTP allows repeated
-                                                              headers.
-                                                            type: array
-                                                            items:
-                                                              description: HTTPHeader
-                                                                describes a custom
-                                                                header to be used
-                                                                in HTTP probes
-                                                              type: object
-                                                              required:
-                                                              - name
-                                                              - value
-                                                              properties:
-                                                                name:
-                                                                  description: The
-                                                                    header field name
-                                                                  type: string
-                                                                value:
-                                                                  description: The
-                                                                    header field value
-                                                                  type: string
-                                                          path:
-                                                            description: Path to access
-                                                              on the HTTP server.
-                                                            type: string
-                                                          port:
-                                                            description: Name or number
-                                                              of the port to access
-                                                              on the container. Number
-                                                              must be in the range
-                                                              1 to 65535. Name must
-                                                              be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                                          scheme:
-                                                            description: Scheme to
-                                                              use for connecting to
-                                                              the host. Defaults to
-                                                              HTTP.
-                                                            type: string
-                                                      tcpSocket:
-                                                        description: 'TCPSocket specifies
-                                                          an action involving a TCP
-                                                          port. TCP hooks not yet
-                                                          supported TODO: implement
-                                                          a realistic TCP lifecycle
-                                                          hook'
-                                                        type: object
-                                                        required:
-                                                        - port
-                                                        properties:
-                                                          host:
-                                                            description: 'Optional:
-                                                              Host name to connect
-                                                              to, defaults to the
-                                                              pod IP.'
-                                                            type: string
-                                                          port:
-                                                            description: Number or
-                                                              name of the port to
-                                                              access on the container.
-                                                              Number must be in the
-                                                              range 1 to 65535. Name
-                                                              must be an IANA_SVC_NAME.
-                                                            anyOf:
-                                                            - type: integer
-                                                            - type: string
-                                                            x-kubernetes-int-or-string: true
-                                              livenessProbe:
-                                                description: 'Periodic probe of container
-                                                  liveness. Container will be restarted
-                                                  if the probe fails. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              name:
-                                                description: Name of the container
-                                                  specified as a DNS_LABEL. Each container
-                                                  in a pod must have a unique name
-                                                  (DNS_LABEL). Cannot be updated.
-                                                type: string
-                                              ports:
-                                                description: List of ports to expose
-                                                  from the container. Exposing a port
-                                                  here gives the system additional
-                                                  information about the network connections
-                                                  a container uses, but is primarily
-                                                  informational. Not specifying a
-                                                  port here DOES NOT prevent that
-                                                  port from being exposed. Any port
-                                                  which is listening on the default
-                                                  "0.0.0.0" address inside a container
-                                                  will be accessible from the network.
-                                                  Cannot be updated.
-                                                type: array
-                                                items:
-                                                  description: ContainerPort represents
-                                                    a network port in a single container.
-                                                  type: object
-                                                  required:
-                                                  - containerPort
-                                                  properties:
-                                                    containerPort:
-                                                      description: Number of port
-                                                        to expose on the pod's IP
-                                                        address. This must be a valid
-                                                        port number, 0 < x < 65536.
-                                                      type: integer
-                                                      format: int32
-                                                    hostIP:
-                                                      description: What host IP to
-                                                        bind the external port to.
-                                                      type: string
-                                                    hostPort:
-                                                      description: Number of port
-                                                        to expose on the host. If
-                                                        specified, this must be a
-                                                        valid port number, 0 < x <
-                                                        65536. If HostNetwork is specified,
-                                                        this must match ContainerPort.
-                                                        Most containers do not need
-                                                        this.
-                                                      type: integer
-                                                      format: int32
-                                                    name:
-                                                      description: If specified, this
-                                                        must be an IANA_SVC_NAME and
-                                                        unique within the pod. Each
-                                                        named port in a pod must have
-                                                        a unique name. Name for the
-                                                        port that can be referred
-                                                        to by services.
-                                                      type: string
-                                                    protocol:
-                                                      description: Protocol for port.
-                                                        Must be UDP, TCP, or SCTP.
-                                                        Defaults to "TCP".
-                                                      type: string
-                                                      default: TCP
-                                                x-kubernetes-list-map-keys:
-                                                - containerPort
-                                                - protocol
-                                                x-kubernetes-list-type: map
-                                              readinessProbe:
-                                                description: 'Periodic probe of container
-                                                  service readiness. Container will
-                                                  be removed from service endpoints
-                                                  if the probe fails. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              resources:
-                                                description: 'Compute Resources required
-                                                  by this container. Cannot be updated.
-                                                  More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                type: object
-                                                properties:
-                                                  limits:
-                                                    description: 'Limits describes
-                                                      the maximum amount of compute
-                                                      resources allowed. More info:
-                                                      https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                                  requests:
-                                                    description: 'Requests describes
-                                                      the minimum amount of compute
-                                                      resources required. If Requests
-                                                      is omitted for a container,
-                                                      it defaults to Limits if that
-                                                      is explicitly specified, otherwise
-                                                      to an implementation-defined
-                                                      value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                                                    type: object
-                                                    additionalProperties:
-                                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                      anyOf:
-                                                      - type: integer
-                                                      - type: string
-                                                      x-kubernetes-int-or-string: true
-                                              securityContext:
-                                                description: 'Security options the
-                                                  pod should run with. More info:
-                                                  https://kubernetes.io/docs/concepts/policy/security-context/
-                                                  More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/'
-                                                type: object
-                                                properties:
-                                                  allowPrivilegeEscalation:
-                                                    description: 'AllowPrivilegeEscalation
-                                                      controls whether a process can
-                                                      gain more privileges than its
-                                                      parent process. This bool directly
-                                                      controls if the no_new_privs
-                                                      flag will be set on the container
-                                                      process. AllowPrivilegeEscalation
-                                                      is true always when the container
-                                                      is: 1) run as Privileged 2)
-                                                      has CAP_SYS_ADMIN'
-                                                    type: boolean
-                                                  capabilities:
-                                                    description: The capabilities
-                                                      to add/drop when running containers.
-                                                      Defaults to the default set
-                                                      of capabilities granted by the
-                                                      container runtime.
-                                                    type: object
-                                                    properties:
-                                                      add:
-                                                        description: Added capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                      drop:
-                                                        description: Removed capabilities
-                                                        type: array
-                                                        items:
-                                                          description: Capability
-                                                            represent POSIX capabilities
-                                                            type
-                                                          type: string
-                                                  privileged:
-                                                    description: Run container in
-                                                      privileged mode. Processes in
-                                                      privileged containers are essentially
-                                                      equivalent to root on the host.
-                                                      Defaults to false.
-                                                    type: boolean
-                                                  procMount:
-                                                    description: procMount denotes
-                                                      the type of proc mount to use
-                                                      for the containers. The default
-                                                      is DefaultProcMount which uses
-                                                      the container runtime defaults
-                                                      for readonly paths and masked
-                                                      paths. This requires the ProcMountType
-                                                      feature flag to be enabled.
-                                                    type: string
-                                                  readOnlyRootFilesystem:
-                                                    description: Whether this container
-                                                      has a read-only root filesystem.
-                                                      Default is false.
-                                                    type: boolean
-                                                  runAsGroup:
-                                                    description: The GID to run the
-                                                      entrypoint of the container
-                                                      process. Uses runtime default
-                                                      if unset. May also be set in
-                                                      PodSecurityContext.  If set
-                                                      in both SecurityContext and
-                                                      PodSecurityContext, the value
-                                                      specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  runAsNonRoot:
-                                                    description: Indicates that the
-                                                      container must run as a non-root
-                                                      user. If true, the Kubelet will
-                                                      validate the image at runtime
-                                                      to ensure that it does not run
-                                                      as UID 0 (root) and fail to
-                                                      start the container if it does.
-                                                      If unset or false, no such validation
-                                                      will be performed. May also
-                                                      be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: boolean
-                                                  runAsUser:
-                                                    description: The UID to run the
-                                                      entrypoint of the container
-                                                      process. Defaults to user specified
-                                                      in image metadata if unspecified.
-                                                      May also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: integer
-                                                    format: int64
-                                                  seLinuxOptions:
-                                                    description: The SELinux context
-                                                      to be applied to the container.
-                                                      If unspecified, the container
-                                                      runtime will allocate a random
-                                                      SELinux context for each container.  May
-                                                      also be set in PodSecurityContext.  If
-                                                      set in both SecurityContext
-                                                      and PodSecurityContext, the
-                                                      value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      level:
-                                                        description: Level is SELinux
-                                                          level label that applies
-                                                          to the container.
-                                                        type: string
-                                                      role:
-                                                        description: Role is a SELinux
-                                                          role label that applies
-                                                          to the container.
-                                                        type: string
-                                                      type:
-                                                        description: Type is a SELinux
-                                                          type label that applies
-                                                          to the container.
-                                                        type: string
-                                                      user:
-                                                        description: User is a SELinux
-                                                          user label that applies
-                                                          to the container.
-                                                        type: string
-                                                  windowsOptions:
-                                                    description: The Windows specific
-                                                      settings applied to all containers.
-                                                      If unspecified, the options
-                                                      from the PodSecurityContext
-                                                      will be used. If set in both
-                                                      SecurityContext and PodSecurityContext,
-                                                      the value specified in SecurityContext
-                                                      takes precedence.
-                                                    type: object
-                                                    properties:
-                                                      gmsaCredentialSpec:
-                                                        description: GMSACredentialSpec
-                                                          is where the GMSA admission
-                                                          webhook (https://github.com/kubernetes-sigs/windows-gmsa)
-                                                          inlines the contents of
-                                                          the GMSA credential spec
-                                                          named by the GMSACredentialSpecName
-                                                          field.
-                                                        type: string
-                                                      gmsaCredentialSpecName:
-                                                        description: GMSACredentialSpecName
-                                                          is the name of the GMSA
-                                                          credential spec to use.
-                                                        type: string
-                                                      runAsUserName:
-                                                        description: The UserName
-                                                          in Windows to run the entrypoint
-                                                          of the container process.
-                                                          Defaults to the user specified
-                                                          in image metadata if unspecified.
-                                                          May also be set in PodSecurityContext.
-                                                          If set in both SecurityContext
-                                                          and PodSecurityContext,
-                                                          the value specified in SecurityContext
-                                                          takes precedence.
-                                                        type: string
-                                              startupProbe:
-                                                description: 'StartupProbe indicates
-                                                  that the Pod has successfully initialized.
-                                                  If specified, no other probes are
-                                                  executed until this completes successfully.
-                                                  If this probe fails, the Pod will
-                                                  be restarted, just as if the livenessProbe
-                                                  failed. This can be used to provide
-                                                  different probe parameters at the
-                                                  beginning of a Pod''s lifecycle,
-                                                  when it might take a long time to
-                                                  load data or warm a cache, than
-                                                  during steady-state operation. This
-                                                  cannot be updated. This is a beta
-                                                  feature enabled by the StartupProbe
-                                                  feature flag. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                type: object
-                                                properties:
-                                                  exec:
-                                                    description: One and only one
-                                                      of the following should be specified.
-                                                      Exec specifies the action to
-                                                      take.
-                                                    type: object
-                                                    properties:
-                                                      command:
-                                                        description: Command is the
-                                                          command line to execute
-                                                          inside the container, the
-                                                          working directory for the
-                                                          command  is root ('/') in
-                                                          the container's filesystem.
-                                                          The command is simply exec'd,
-                                                          it is not run inside a shell,
-                                                          so traditional shell instructions
-                                                          ('|', etc) won't work. To
-                                                          use a shell, you need to
-                                                          explicitly call out to that
-                                                          shell. Exit status of 0
-                                                          is treated as live/healthy
-                                                          and non-zero is unhealthy.
-                                                        type: array
-                                                        items:
-                                                          type: string
-                                                  failureThreshold:
-                                                    description: Minimum consecutive
-                                                      failures for the probe to be
-                                                      considered failed after having
-                                                      succeeded. Defaults to 3. Minimum
-                                                      value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  httpGet:
-                                                    description: HTTPGet specifies
-                                                      the http request to perform.
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: Host name to
-                                                          connect to, defaults to
-                                                          the pod IP. You probably
-                                                          want to set "Host" in httpHeaders
-                                                          instead.
-                                                        type: string
-                                                      httpHeaders:
-                                                        description: Custom headers
-                                                          to set in the request. HTTP
-                                                          allows repeated headers.
-                                                        type: array
-                                                        items:
-                                                          description: HTTPHeader
-                                                            describes a custom header
-                                                            to be used in HTTP probes
-                                                          type: object
-                                                          required:
-                                                          - name
-                                                          - value
-                                                          properties:
-                                                            name:
-                                                              description: The header
-                                                                field name
-                                                              type: string
-                                                            value:
-                                                              description: The header
-                                                                field value
-                                                              type: string
-                                                      path:
-                                                        description: Path to access
-                                                          on the HTTP server.
-                                                        type: string
-                                                      port:
-                                                        description: Name or number
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                      scheme:
-                                                        description: Scheme to use
-                                                          for connecting to the host.
-                                                          Defaults to HTTP.
-                                                        type: string
-                                                  initialDelaySeconds:
-                                                    description: 'Number of seconds
-                                                      after the container has started
-                                                      before liveness probes are initiated.
-                                                      More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                                  periodSeconds:
-                                                    description: How often (in seconds)
-                                                      to perform the probe. Default
-                                                      to 10 seconds. Minimum value
-                                                      is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  successThreshold:
-                                                    description: Minimum consecutive
-                                                      successes for the probe to be
-                                                      considered successful after
-                                                      having failed. Defaults to 1.
-                                                      Must be 1 for liveness and startup.
-                                                      Minimum value is 1.
-                                                    type: integer
-                                                    format: int32
-                                                  tcpSocket:
-                                                    description: 'TCPSocket specifies
-                                                      an action involving a TCP port.
-                                                      TCP hooks not yet supported
-                                                      TODO: implement a realistic
-                                                      TCP lifecycle hook'
-                                                    type: object
-                                                    required:
-                                                    - port
-                                                    properties:
-                                                      host:
-                                                        description: 'Optional: Host
-                                                          name to connect to, defaults
-                                                          to the pod IP.'
-                                                        type: string
-                                                      port:
-                                                        description: Number or name
-                                                          of the port to access on
-                                                          the container. Number must
-                                                          be in the range 1 to 65535.
-                                                          Name must be an IANA_SVC_NAME.
-                                                        anyOf:
-                                                        - type: integer
-                                                        - type: string
-                                                        x-kubernetes-int-or-string: true
-                                                  timeoutSeconds:
-                                                    description: 'Number of seconds
-                                                      after which the probe times
-                                                      out. Defaults to 1 second. Minimum
-                                                      value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
-                                                    type: integer
-                                                    format: int32
-                                              stdin:
-                                                description: Whether this container
-                                                  should allocate a buffer for stdin
-                                                  in the container runtime. If this
-                                                  is not set, reads from stdin in
-                                                  the container will always result
-                                                  in EOF. Default is false.
-                                                type: boolean
-                                              stdinOnce:
-                                                description: Whether the container
-                                                  runtime should close the stdin channel
-                                                  after it has been opened by a single
-                                                  attach. When stdin is true the stdin
-                                                  stream will remain open across multiple
-                                                  attach sessions. If stdinOnce is
-                                                  set to true, stdin is opened on
-                                                  container start, is empty until
-                                                  the first client attaches to stdin,
-                                                  and then remains open and accepts
-                                                  data until the client disconnects,
-                                                  at which time stdin is closed and
-                                                  remains closed until the container
-                                                  is restarted. If this flag is false,
-                                                  a container processes that reads
-                                                  from stdin will never receive an
-                                                  EOF. Default is false
-                                                type: boolean
-                                              terminationMessagePath:
-                                                description: 'Optional: Path at which
-                                                  the file to which the container''s
-                                                  termination message will be written
-                                                  is mounted into the container''s
-                                                  filesystem. Message written is intended
-                                                  to be brief final status, such as
-                                                  an assertion failure message. Will
-                                                  be truncated by the node if greater
-                                                  than 4096 bytes. The total message
-                                                  length across all containers will
-                                                  be limited to 12kb. Defaults to
-                                                  /dev/termination-log. Cannot be
-                                                  updated.'
-                                                type: string
-                                              terminationMessagePolicy:
-                                                description: Indicate how the termination
-                                                  message should be populated. File
-                                                  will use the contents of terminationMessagePath
-                                                  to populate the container status
-                                                  message on both success and failure.
-                                                  FallbackToLogsOnError will use the
-                                                  last chunk of container log output
-                                                  if the termination message file
-                                                  is empty and the container exited
-                                                  with an error. The log output is
-                                                  limited to 2048 bytes or 80 lines,
-                                                  whichever is smaller. Defaults to
-                                                  File. Cannot be updated.
-                                                type: string
-                                              tty:
-                                                description: Whether this container
-                                                  should allocate a TTY for itself,
-                                                  also requires 'stdin' to be true.
-                                                  Default is false.
-                                                type: boolean
-                                              volumeDevices:
-                                                description: volumeDevices is the
-                                                  list of block devices to be used
-                                                  by the container.
-                                                type: array
-                                                items:
-                                                  description: volumeDevice describes
-                                                    a mapping of a raw block device
-                                                    within a container.
-                                                  type: object
-                                                  required:
-                                                  - devicePath
-                                                  - name
-                                                  properties:
-                                                    devicePath:
-                                                      description: devicePath is the
-                                                        path inside of the container
-                                                        that the device will be mapped
-                                                        to.
-                                                      type: string
-                                                    name:
-                                                      description: name must match
-                                                        the name of a persistentVolumeClaim
-                                                        in the pod
-                                                      type: string
-                                              volumeMounts:
-                                                description: Pod volumes to mount
-                                                  into the container's filesystem.
-                                                  Cannot be updated.
-                                                type: array
-                                                items:
-                                                  description: VolumeMount describes
-                                                    a mounting of a Volume within
-                                                    a container.
-                                                  type: object
-                                                  required:
-                                                  - mountPath
-                                                  - name
-                                                  properties:
-                                                    mountPath:
-                                                      description: Path within the
-                                                        container at which the volume
-                                                        should be mounted.  Must not
-                                                        contain ':'.
-                                                      type: string
-                                                    mountPropagation:
-                                                      description: mountPropagation
-                                                        determines how mounts are
-                                                        propagated from the host to
-                                                        container and the other way
-                                                        around. When not set, MountPropagationNone
-                                                        is used. This field is beta
-                                                        in 1.10.
-                                                      type: string
-                                                    name:
-                                                      description: This must match
-                                                        the Name of a Volume.
-                                                      type: string
-                                                    readOnly:
-                                                      description: Mounted read-only
-                                                        if true, read-write otherwise
-                                                        (false or unspecified). Defaults
-                                                        to false.
-                                                      type: boolean
-                                                    subPath:
-                                                      description: Path within the
-                                                        volume from which the container's
-                                                        volume should be mounted.
-                                                        Defaults to "" (volume's root).
-                                                      type: string
-                                                    subPathExpr:
-                                                      description: Expanded path within
-                                                        the volume from which the
-                                                        container's volume should
-                                                        be mounted. Behaves similarly
-                                                        to SubPath but environment
-                                                        variable references $(VAR_NAME)
-                                                        are expanded using the container's
-                                                        environment. Defaults to ""
-                                                        (volume's root). SubPathExpr
-                                                        and SubPath are mutually exclusive.
-                                                      type: string
-                                              workingDir:
-                                                description: Container's working directory.
-                                                  If not specified, the container
-                                                  runtime's default will be used,
-                                                  which might be configured in the
-                                                  container image. Cannot be updated.
-                                                type: string
-                                        nodeName:
-                                          description: NodeName is a request to schedule
-                                            this pod onto a specific node. If it is
-                                            non-empty, the scheduler simply schedules
-                                            this pod onto that node, assuming that
-                                            it fits resource requirements.
-                                          type: string
-                                        nodeSelector:
-                                          description: 'NodeSelector is a selector
-                                            which must be true for the pod to fit
-                                            on a node. Selector which must match a
-                                            node''s labels for the pod to be scheduled
-                                            on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/'
-                                          type: object
-                                          additionalProperties:
-                                            type: string
-                                        overhead:
-                                          description: 'Overhead represents the resource
-                                            overhead associated with running a pod
-                                            for a given RuntimeClass. This field will
-                                            be autopopulated at admission time by
-                                            the RuntimeClass admission controller.
-                                            If the RuntimeClass admission controller
-                                            is enabled, overhead must not be set in
-                                            Pod create requests. The RuntimeClass
-                                            admission controller will reject Pod create
-                                            requests which have the overhead already
-                                            set. If RuntimeClass is configured and
-                                            selected in the PodSpec, Overhead will
-                                            be set to the value defined in the corresponding
-                                            RuntimeClass, otherwise it will remain
-                                            unset and treated as zero. More info:
-                                            https://git.k8s.io/enhancements/keps/sig-node/20190226-pod-overhead.md
-                                            This field is alpha-level as of Kubernetes
-                                            v1.16, and is only honored by servers
-                                            that enable the PodOverhead feature.'
-                                          type: object
-                                          additionalProperties:
-                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                            anyOf:
-                                            - type: integer
-                                            - type: string
-                                            x-kubernetes-int-or-string: true
-                                        preemptionPolicy:
-                                          description: PreemptionPolicy is the Policy
-                                            for preempting pods with lower priority.
-                                            One of Never, PreemptLowerPriority. Defaults
-                                            to PreemptLowerPriority if unset. This
-                                            field is alpha-level and is only honored
-                                            by servers that enable the NonPreemptingPriority
-                                            feature.
-                                          type: string
-                                        priority:
-                                          description: The priority value. Various
-                                            system components use this field to find
-                                            the priority of the pod. When Priority
-                                            Admission Controller is enabled, it prevents
-                                            users from setting this field. The admission
-                                            controller populates this field from PriorityClassName.
-                                            The higher the value, the higher the priority.
-                                          type: integer
-                                          format: int32
-                                        priorityClassName:
-                                          description: If specified, indicates the
-                                            pod's priority. "system-node-critical"
-                                            and "system-cluster-critical" are two
-                                            special keywords which indicate the highest
-                                            priorities with the former being the highest
-                                            priority. Any other name must be defined
-                                            by creating a PriorityClass object with
-                                            that name. If not specified, the pod priority
-                                            will be default or zero if there is no
-                                            default.
-                                          type: string
-                                        readinessGates:
-                                          description: 'If specified, all readiness
-                                            gates will be evaluated for pod readiness.
-                                            A pod is ready when all its containers
-                                            are ready AND all conditions specified
-                                            in the readiness gates have status equal
-                                            to "True" More info: https://git.k8s.io/enhancements/keps/sig-network/0007-pod-ready%2B%2B.md'
-                                          type: array
-                                          items:
-                                            description: PodReadinessGate contains
-                                              the reference to a pod condition
-                                            type: object
-                                            required:
-                                            - conditionType
-                                            properties:
-                                              conditionType:
-                                                description: ConditionType refers
-                                                  to a condition in the pod's condition
-                                                  list with matching type.
-                                                type: string
-                                        restartPolicy:
-                                          description: 'Restart policy for all containers
-                                            within the pod. One of Always, OnFailure,
-                                            Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy'
-                                          type: string
-                                        runtimeClassName:
-                                          description: 'RuntimeClassName refers to
-                                            a RuntimeClass object in the node.k8s.io
-                                            group, which should be used to run this
-                                            pod.  If no RuntimeClass resource matches
-                                            the named class, the pod will not be run.
-                                            If unset or empty, the "legacy" RuntimeClass
-                                            will be used, which is an implicit class
-                                            with an empty definition that uses the
-                                            default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/runtime-class.md
-                                            This is a beta feature as of Kubernetes
-                                            v1.14.'
-                                          type: string
-                                        schedulerName:
-                                          description: If specified, the pod will
-                                            be dispatched by specified scheduler.
-                                            If not specified, the pod will be dispatched
-                                            by default scheduler.
-                                          type: string
-                                        securityContext:
-                                          description: 'SecurityContext holds pod-level
-                                            security attributes and common container
-                                            settings. Optional: Defaults to empty.  See
-                                            type description for default values of
-                                            each field.'
-                                          type: object
-                                          properties:
-                                            fsGroup:
-                                              description: "A special supplemental
-                                                group that applies to all containers
-                                                in a pod. Some volume types allow
-                                                the Kubelet to change the ownership
-                                                of that volume to be owned by the
-                                                pod: \n 1. The owning GID will be
-                                                the FSGroup 2. The setgid bit is set
-                                                (new files created in the volume will
-                                                be owned by FSGroup) 3. The permission
-                                                bits are OR'd with rw-rw---- \n If
-                                                unset, the Kubelet will not modify
-                                                the ownership and permissions of any
-                                                volume."
-                                              type: integer
-                                              format: int64
-                                            fsGroupChangePolicy:
-                                              description: 'fsGroupChangePolicy defines
-                                                behavior of changing ownership and
-                                                permission of the volume before being
-                                                exposed inside Pod. This field will
-                                                only apply to volume types which support
-                                                fsGroup based ownership(and permissions).
-                                                It will have no effect on ephemeral
-                                                volume types such as: secret, configmaps
-                                                and emptydir. Valid values are "OnRootMismatch"
-                                                and "Always". If not specified defaults
-                                                to "Always".'
-                                              type: string
-                                            runAsGroup:
-                                              description: The GID to run the entrypoint
-                                                of the container process. Uses runtime
-                                                default if unset. May also be set
-                                                in SecurityContext.  If set in both
-                                                SecurityContext and PodSecurityContext,
-                                                the value specified in SecurityContext
-                                                takes precedence for that container.
-                                              type: integer
-                                              format: int64
-                                            runAsNonRoot:
-                                              description: Indicates that the container
-                                                must run as a non-root user. If true,
-                                                the Kubelet will validate the image
-                                                at runtime to ensure that it does
-                                                not run as UID 0 (root) and fail to
-                                                start the container if it does. If
-                                                unset or false, no such validation
-                                                will be performed. May also be set
-                                                in SecurityContext.  If set in both
-                                                SecurityContext and PodSecurityContext,
-                                                the value specified in SecurityContext
-                                                takes precedence.
-                                              type: boolean
-                                            runAsUser:
-                                              description: The UID to run the entrypoint
-                                                of the container process. Defaults
-                                                to user specified in image metadata
-                                                if unspecified. May also be set in
-                                                SecurityContext.  If set in both SecurityContext
-                                                and PodSecurityContext, the value
-                                                specified in SecurityContext takes
-                                                precedence for that container.
-                                              type: integer
-                                              format: int64
-                                            seLinuxOptions:
-                                              description: The SELinux context to
-                                                be applied to all containers. If unspecified,
-                                                the container runtime will allocate
-                                                a random SELinux context for each
-                                                container.  May also be set in SecurityContext.  If
-                                                set in both SecurityContext and PodSecurityContext,
-                                                the value specified in SecurityContext
-                                                takes precedence for that container.
-                                              type: object
-                                              properties:
-                                                level:
-                                                  description: Level is SELinux level
-                                                    label that applies to the container.
-                                                  type: string
-                                                role:
-                                                  description: Role is a SELinux role
-                                                    label that applies to the container.
-                                                  type: string
-                                                type:
-                                                  description: Type is a SELinux type
-                                                    label that applies to the container.
-                                                  type: string
-                                                user:
-                                                  description: User is a SELinux user
-                                                    label that applies to the container.
-                                                  type: string
-                                            supplementalGroups:
-                                              description: A list of groups applied
-                                                to the first process run in each container,
-                                                in addition to the container's primary
-                                                GID.  If unspecified, no groups will
-                                                be added to any container.
-                                              type: array
-                                              items:
-                                                type: integer
-                                                format: int64
-                                            sysctls:
-                                              description: Sysctls hold a list of
-                                                namespaced sysctls used for the pod.
-                                                Pods with unsupported sysctls (by
-                                                the container runtime) might fail
-                                                to launch.
-                                              type: array
-                                              items:
-                                                description: Sysctl defines a kernel
-                                                  parameter to be set
-                                                type: object
-                                                required:
-                                                - name
-                                                - value
-                                                properties:
-                                                  name:
-                                                    description: Name of a property
-                                                      to set
-                                                    type: string
-                                                  value:
-                                                    description: Value of a property
-                                                      to set
-                                                    type: string
-                                            windowsOptions:
-                                              description: The Windows specific settings
-                                                applied to all containers. If unspecified,
-                                                the options within a container's SecurityContext
-                                                will be used. If set in both SecurityContext
-                                                and PodSecurityContext, the value
-                                                specified in SecurityContext takes
-                                                precedence.
-                                              type: object
-                                              properties:
-                                                gmsaCredentialSpec:
-                                                  description: GMSACredentialSpec
-                                                    is where the GMSA admission webhook
-                                                    (https://github.com/kubernetes-sigs/windows-gmsa)
-                                                    inlines the contents of the GMSA
-                                                    credential spec named by the GMSACredentialSpecName
-                                                    field.
-                                                  type: string
-                                                gmsaCredentialSpecName:
-                                                  description: GMSACredentialSpecName
-                                                    is the name of the GMSA credential
-                                                    spec to use.
-                                                  type: string
-                                                runAsUserName:
-                                                  description: The UserName in Windows
-                                                    to run the entrypoint of the container
-                                                    process. Defaults to the user
-                                                    specified in image metadata if
-                                                    unspecified. May also be set in
-                                                    PodSecurityContext. If set in
-                                                    both SecurityContext and PodSecurityContext,
-                                                    the value specified in SecurityContext
-                                                    takes precedence.
-                                                  type: string
-                                        serviceAccount:
-                                          description: 'DeprecatedServiceAccount is
-                                            a depreciated alias for ServiceAccountName.
-                                            Deprecated: Use serviceAccountName instead.'
-                                          type: string
-                                        serviceAccountName:
-                                          description: 'ServiceAccountName is the
-                                            name of the ServiceAccount to use to run
-                                            this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/'
-                                          type: string
-                                        shareProcessNamespace:
-                                          description: 'Share a single process namespace
-                                            between all of the containers in a pod.
-                                            When this is set containers will be able
-                                            to view and signal processes from other
-                                            containers in the same pod, and the first
-                                            process in each container will not be
-                                            assigned PID 1. HostPID and ShareProcessNamespace
-                                            cannot both be set. Optional: Default
-                                            to false.'
-                                          type: boolean
-                                        subdomain:
-                                          description: If specified, the fully qualified
-                                            Pod hostname will be "<hostname>.<subdomain>.<pod
-                                            namespace>.svc.<cluster domain>". If not
-                                            specified, the pod will not have a domainname
-                                            at all.
-                                          type: string
-                                        terminationGracePeriodSeconds:
-                                          description: Optional duration in seconds
-                                            the pod needs to terminate gracefully.
-                                            May be decreased in delete request. Value
-                                            must be non-negative integer. The value
-                                            zero indicates delete immediately. If
-                                            this value is nil, the default grace period
-                                            will be used instead. The grace period
-                                            is the duration in seconds after the processes
-                                            running in the pod are sent a termination
-                                            signal and the time when the processes
-                                            are forcibly halted with a kill signal.
-                                            Set this value longer than the expected
-                                            cleanup time for your process. Defaults
-                                            to 30 seconds.
-                                          type: integer
-                                          format: int64
-                                        tolerations:
-                                          description: If specified, the pod's tolerations.
-                                          type: array
-                                          items:
-                                            description: The pod this Toleration is
-                                              attached to tolerates any taint that
-                                              matches the triple <key,value,effect>
-                                              using the matching operator <operator>.
-                                            type: object
-                                            properties:
-                                              effect:
-                                                description: Effect indicates the
-                                                  taint effect to match. Empty means
-                                                  match all taint effects. When specified,
-                                                  allowed values are NoSchedule, PreferNoSchedule
-                                                  and NoExecute.
-                                                type: string
-                                              key:
-                                                description: Key is the taint key
-                                                  that the toleration applies to.
-                                                  Empty means match all taint keys.
-                                                  If the key is empty, operator must
-                                                  be Exists; this combination means
-                                                  to match all values and all keys.
-                                                type: string
-                                              operator:
-                                                description: Operator represents a
-                                                  key's relationship to the value.
-                                                  Valid operators are Exists and Equal.
-                                                  Defaults to Equal. Exists is equivalent
-                                                  to wildcard for value, so that a
-                                                  pod can tolerate all taints of a
-                                                  particular category.
-                                                type: string
-                                              tolerationSeconds:
-                                                description: TolerationSeconds represents
-                                                  the period of time the toleration
-                                                  (which must be of effect NoExecute,
-                                                  otherwise this field is ignored)
-                                                  tolerates the taint. By default,
-                                                  it is not set, which means tolerate
-                                                  the taint forever (do not evict).
-                                                  Zero and negative values will be
-                                                  treated as 0 (evict immediately)
-                                                  by the system.
-                                                type: integer
-                                                format: int64
-                                              value:
-                                                description: Value is the taint value
-                                                  the toleration matches to. If the
-                                                  operator is Exists, the value should
-                                                  be empty, otherwise just a regular
-                                                  string.
-                                                type: string
-                                        topologySpreadConstraints:
-                                          description: TopologySpreadConstraints describes
-                                            how a group of pods ought to spread across
-                                            topology domains. Scheduler will schedule
-                                            pods in a way which abides by the constraints.
-                                            This field is only honored by clusters
-                                            that enable the EvenPodsSpread feature.
-                                            All topologySpreadConstraints are ANDed.
-                                          type: array
-                                          items:
-                                            description: TopologySpreadConstraint
-                                              specifies how to spread matching pods
-                                              among the given topology.
-                                            type: object
-                                            required:
-                                            - maxSkew
-                                            - topologyKey
-                                            - whenUnsatisfiable
-                                            properties:
-                                              labelSelector:
-                                                description: LabelSelector is used
-                                                  to find matching pods. Pods that
-                                                  match this label selector are counted
-                                                  to determine the number of pods
-                                                  in their corresponding topology
-                                                  domain.
-                                                type: object
-                                                properties:
-                                                  matchExpressions:
-                                                    description: matchExpressions
-                                                      is a list of label selector
-                                                      requirements. The requirements
-                                                      are ANDed.
-                                                    type: array
-                                                    items:
-                                                      description: A label selector
-                                                        requirement is a selector
-                                                        that contains values, a key,
-                                                        and an operator that relates
-                                                        the key and values.
-                                                      type: object
-                                                      required:
-                                                      - key
-                                                      - operator
-                                                      properties:
-                                                        key:
-                                                          description: key is the
-                                                            label key that the selector
-                                                            applies to.
-                                                          type: string
-                                                        operator:
-                                                          description: operator represents
-                                                            a key's relationship to
-                                                            a set of values. Valid
-                                                            operators are In, NotIn,
-                                                            Exists and DoesNotExist.
-                                                          type: string
-                                                        values:
-                                                          description: values is an
-                                                            array of string values.
-                                                            If the operator is In
-                                                            or NotIn, the values array
-                                                            must be non-empty. If
-                                                            the operator is Exists
-                                                            or DoesNotExist, the values
-                                                            array must be empty. This
-                                                            array is replaced during
-                                                            a strategic merge patch.
-                                                          type: array
-                                                          items:
-                                                            type: string
-                                                  matchLabels:
-                                                    description: matchLabels is a
-                                                      map of {key,value} pairs. A
-                                                      single {key,value} in the matchLabels
-                                                      map is equivalent to an element
-                                                      of matchExpressions, whose key
-                                                      field is "key", the operator
-                                                      is "In", and the values array
-                                                      contains only "value". The requirements
-                                                      are ANDed.
-                                                    type: object
-                                                    additionalProperties:
-                                                      type: string
-                                              maxSkew:
-                                                description: 'MaxSkew describes the
-                                                  degree to which pods may be unevenly
-                                                  distributed. It''s the maximum permitted
-                                                  difference between the number of
-                                                  matching pods in any two topology
-                                                  domains of a given topology type.
-                                                  For example, in a 3-zone cluster,
-                                                  MaxSkew is set to 1, and pods with
-                                                  the same labelSelector spread as
-                                                  1/1/0: | zone1 | zone2 | zone3 |
-                                                  |   P   |   P   |       | - if MaxSkew
-                                                  is 1, incoming pod can only be scheduled
-                                                  to zone3 to become 1/1/1; scheduling
-                                                  it onto zone1(zone2) would make
-                                                  the ActualSkew(2-0) on zone1(zone2)
-                                                  violate MaxSkew(1). - if MaxSkew
-                                                  is 2, incoming pod can be scheduled
-                                                  onto any zone. It''s a required
-                                                  field. Default value is 1 and 0
-                                                  is not allowed.'
-                                                type: integer
-                                                format: int32
-                                              topologyKey:
-                                                description: TopologyKey is the key
-                                                  of node labels. Nodes that have
-                                                  a label with this key and identical
-                                                  values are considered to be in the
-                                                  same topology. We consider each
-                                                  <key, value> as a "bucket", and
-                                                  try to put balanced number of pods
-                                                  into each bucket. It's a required
-                                                  field.
-                                                type: string
-                                              whenUnsatisfiable:
-                                                description: 'WhenUnsatisfiable indicates
-                                                  how to deal with a pod if it doesn''t
-                                                  satisfy the spread constraint. -
-                                                  DoNotSchedule (default) tells the
-                                                  scheduler not to schedule it - ScheduleAnyway
-                                                  tells the scheduler to still schedule
-                                                  it It''s considered as "Unsatisfiable"
-                                                  if and only if placing incoming
-                                                  pod on any topology violates "MaxSkew".
-                                                  For example, in a 3-zone cluster,
-                                                  MaxSkew is set to 1, and pods with
-                                                  the same labelSelector spread as
-                                                  3/1/1: | zone1 | zone2 | zone3 |
-                                                  | P P P |   P   |   P   | If WhenUnsatisfiable
-                                                  is set to DoNotSchedule, incoming
-                                                  pod can only be scheduled to zone2(zone3)
-                                                  to become 3/2/1(3/1/2) as ActualSkew(2-1)
-                                                  on zone2(zone3) satisfies MaxSkew(1).
-                                                  In other words, the cluster can
-                                                  still be imbalanced, but scheduler
-                                                  won''t make it *more* imbalanced.
-                                                  It''s a required field.'
-                                                type: string
-                                          x-kubernetes-list-map-keys:
-                                          - topologyKey
-                                          - whenUnsatisfiable
-                                          x-kubernetes-list-type: map
-                                        volumes:
-                                          description: 'List of volumes that can be
-                                            mounted by containers belonging to the
-                                            pod. More info: https://kubernetes.io/docs/concepts/storage/volumes'
-                                          type: array
-                                          items:
-                                            description: Volume represents a named
-                                              volume in a pod that may be accessed
-                                              by any container in the pod.
-                                            type: object
-                                            required:
-                                            - name
-                                            properties:
-                                              awsElasticBlockStore:
-                                                description: 'AWSElasticBlockStore
-                                                  represents an AWS Disk resource
-                                                  that is attached to a kubelet''s
-                                                  host machine and then exposed to
-                                                  the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                                                type: object
-                                                required:
-                                                - volumeID
-                                                properties:
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      of the volume that you want
-                                                      to mount. Tip: Ensure that the
-                                                      filesystem type is supported
-                                                      by the host operating system.
-                                                      Examples: "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
-                                                      TODO: how do we prevent errors
-                                                      in the filesystem from compromising
-                                                      the machine'
-                                                    type: string
-                                                  partition:
-                                                    description: 'The partition in
-                                                      the volume that you want to
-                                                      mount. If omitted, the default
-                                                      is to mount by volume name.
-                                                      Examples: For volume /dev/sda1,
-                                                      you specify the partition as
-                                                      "1". Similarly, the volume partition
-                                                      for /dev/sda is "0" (or you
-                                                      can leave the property empty).'
-                                                    type: integer
-                                                    format: int32
-                                                  readOnly:
-                                                    description: 'Specify "true" to
-                                                      force and set the ReadOnly property
-                                                      in VolumeMounts to "true". If
-                                                      omitted, the default is "false".
-                                                      More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                                                    type: boolean
-                                                  volumeID:
-                                                    description: 'Unique ID of the
-                                                      persistent disk resource in
-                                                      AWS (Amazon EBS volume). More
-                                                      info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                                                    type: string
-                                              azureDisk:
-                                                description: AzureDisk represents
-                                                  an Azure Data Disk mount on the
-                                                  host and bind mount to the pod.
-                                                type: object
-                                                required:
-                                                - diskName
-                                                - diskURI
-                                                properties:
-                                                  cachingMode:
-                                                    description: 'Host Caching mode:
-                                                      None, Read Only, Read Write.'
-                                                    type: string
-                                                  diskName:
-                                                    description: The Name of the data
-                                                      disk in the blob storage
-                                                    type: string
-                                                  diskURI:
-                                                    description: The URI the data
-                                                      disk in the blob storage
-                                                    type: string
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified.
-                                                    type: string
-                                                  kind:
-                                                    description: 'Expected values
-                                                      Shared: multiple blob disks
-                                                      per storage account  Dedicated:
-                                                      single blob disk per storage
-                                                      account  Managed: azure managed
-                                                      data disk (only in managed availability
-                                                      set). defaults to shared'
-                                                    type: string
-                                                  readOnly:
-                                                    description: Defaults to false
-                                                      (read/write). ReadOnly here
-                                                      will force the ReadOnly setting
-                                                      in VolumeMounts.
-                                                    type: boolean
-                                              azureFile:
-                                                description: AzureFile represents
-                                                  an Azure File Service mount on the
-                                                  host and bind mount to the pod.
-                                                type: object
-                                                required:
-                                                - secretName
-                                                - shareName
-                                                properties:
-                                                  readOnly:
-                                                    description: Defaults to false
-                                                      (read/write). ReadOnly here
-                                                      will force the ReadOnly setting
-                                                      in VolumeMounts.
-                                                    type: boolean
-                                                  secretName:
-                                                    description: the name of secret
-                                                      that contains Azure Storage
-                                                      Account Name and Key
-                                                    type: string
-                                                  shareName:
-                                                    description: Share Name
-                                                    type: string
-                                              cephfs:
-                                                description: CephFS represents a Ceph
-                                                  FS mount on the host that shares
-                                                  a pod's lifetime
-                                                type: object
-                                                required:
-                                                - monitors
-                                                properties:
-                                                  monitors:
-                                                    description: 'Required: Monitors
-                                                      is a collection of Ceph monitors
-                                                      More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                                                    type: array
-                                                    items:
-                                                      type: string
-                                                  path:
-                                                    description: 'Optional: Used as
-                                                      the mounted root, rather than
-                                                      the full Ceph tree, default
-                                                      is /'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'Optional: Defaults
-                                                      to false (read/write). ReadOnly
-                                                      here will force the ReadOnly
-                                                      setting in VolumeMounts. More
-                                                      info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                                                    type: boolean
-                                                  secretFile:
-                                                    description: 'Optional: SecretFile
-                                                      is the path to key ring for
-                                                      User, default is /etc/ceph/user.secret
-                                                      More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                                                    type: string
-                                                  secretRef:
-                                                    description: 'Optional: SecretRef
-                                                      is reference to the authentication
-                                                      secret for User, default is
-                                                      empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  user:
-                                                    description: 'Optional: User is
-                                                      the rados user name, default
-                                                      is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                                                    type: string
-                                              cinder:
-                                                description: 'Cinder represents a
-                                                  cinder volume attached and mounted
-                                                  on kubelets host machine. More info:
-                                                  https://examples.k8s.io/mysql-cinder-pd/README.md'
-                                                type: object
-                                                required:
-                                                - volumeID
-                                                properties:
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      to mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Examples: "ext4", "xfs",
-                                                      "ntfs". Implicitly inferred
-                                                      to be "ext4" if unspecified.
-                                                      More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'Optional: Defaults
-                                                      to false (read/write). ReadOnly
-                                                      here will force the ReadOnly
-                                                      setting in VolumeMounts. More
-                                                      info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: 'Optional: points
-                                                      to a secret object containing
-                                                      parameters used to connect to
-                                                      OpenStack.'
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  volumeID:
-                                                    description: 'volume id used to
-                                                      identify the volume in cinder.
-                                                      More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                                                    type: string
-                                              configMap:
-                                                description: ConfigMap represents
-                                                  a configMap that should populate
-                                                  this volume
-                                                type: object
-                                                properties:
-                                                  defaultMode:
-                                                    description: 'Optional: mode bits
-                                                      to use on created files by default.
-                                                      Must be a value between 0 and
-                                                      0777. Defaults to 0644. Directories
-                                                      within the path are not affected
-                                                      by this setting. This might
-                                                      be in conflict with other options
-                                                      that affect the file mode, like
-                                                      fsGroup, and the result can
-                                                      be other mode bits set.'
-                                                    type: integer
-                                                    format: int32
-                                                  items:
-                                                    description: If unspecified, each
-                                                      key-value pair in the Data field
-                                                      of the referenced ConfigMap
-                                                      will be projected into the volume
-                                                      as a file whose name is the
-                                                      key and content is the value.
-                                                      If specified, the listed keys
-                                                      will be projected into the specified
-                                                      paths, and unlisted keys will
-                                                      not be present. If a key is
-                                                      specified which is not present
-                                                      in the ConfigMap, the volume
-                                                      setup will error unless it is
-                                                      marked optional. Paths must
-                                                      be relative and may not contain
-                                                      the '..' path or start with
-                                                      '..'.
-                                                    type: array
-                                                    items:
-                                                      description: Maps a string key
-                                                        to a path within a volume.
-                                                      type: object
-                                                      required:
-                                                      - key
-                                                      - path
-                                                      properties:
-                                                        key:
-                                                          description: The key to
-                                                            project.
-                                                          type: string
-                                                        mode:
-                                                          description: 'Optional:
-                                                            mode bits to use on this
-                                                            file, must be a value
-                                                            between 0 and 0777. If
-                                                            not specified, the volume
-                                                            defaultMode will be used.
-                                                            This might be in conflict
-                                                            with other options that
-                                                            affect the file mode,
-                                                            like fsGroup, and the
-                                                            result can be other mode
-                                                            bits set.'
-                                                          type: integer
-                                                          format: int32
-                                                        path:
-                                                          description: The relative
-                                                            path of the file to map
-                                                            the key to. May not be
-                                                            an absolute path. May
-                                                            not contain the path element
-                                                            '..'. May not start with
-                                                            the string '..'.
-                                                          type: string
-                                                  name:
-                                                    description: 'Name of the referent.
-                                                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                      TODO: Add other useful fields.
-                                                      apiVersion, kind, uid?'
-                                                    type: string
-                                                  optional:
-                                                    description: Specify whether the
-                                                      ConfigMap or its keys must be
-                                                      defined
-                                                    type: boolean
-                                              csi:
-                                                description: CSI (Container Storage
-                                                  Interface) represents storage that
-                                                  is handled by an external CSI driver
-                                                  (Alpha feature).
-                                                type: object
-                                                required:
-                                                - driver
-                                                properties:
-                                                  driver:
-                                                    description: Driver is the name
-                                                      of the CSI driver that handles
-                                                      this volume. Consult with your
-                                                      admin for the correct name as
-                                                      registered in the cluster.
-                                                    type: string
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Ex. "ext4", "xfs", "ntfs".
-                                                      If not provided, the empty value
-                                                      is passed to the associated
-                                                      CSI driver which will determine
-                                                      the default filesystem to apply.
-                                                    type: string
-                                                  nodePublishSecretRef:
-                                                    description: NodePublishSecretRef
-                                                      is a reference to the secret
-                                                      object containing sensitive
-                                                      information to pass to the CSI
-                                                      driver to complete the CSI NodePublishVolume
-                                                      and NodeUnpublishVolume calls.
-                                                      This field is optional, and  may
-                                                      be empty if no secret is required.
-                                                      If the secret object contains
-                                                      more than one secret, all secret
-                                                      references are passed.
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  readOnly:
-                                                    description: Specifies a read-only
-                                                      configuration for the volume.
-                                                      Defaults to false (read/write).
-                                                    type: boolean
-                                                  volumeAttributes:
-                                                    description: VolumeAttributes
-                                                      stores driver-specific properties
-                                                      that are passed to the CSI driver.
-                                                      Consult your driver's documentation
-                                                      for supported values.
-                                                    type: object
-                                                    additionalProperties:
-                                                      type: string
-                                              downwardAPI:
-                                                description: DownwardAPI represents
-                                                  downward API about the pod that
-                                                  should populate this volume
-                                                type: object
-                                                properties:
-                                                  defaultMode:
-                                                    description: 'Optional: mode bits
-                                                      to use on created files by default.
-                                                      Must be a value between 0 and
-                                                      0777. Defaults to 0644. Directories
-                                                      within the path are not affected
-                                                      by this setting. This might
-                                                      be in conflict with other options
-                                                      that affect the file mode, like
-                                                      fsGroup, and the result can
-                                                      be other mode bits set.'
-                                                    type: integer
-                                                    format: int32
-                                                  items:
-                                                    description: Items is a list of
-                                                      downward API volume file
-                                                    type: array
-                                                    items:
-                                                      description: DownwardAPIVolumeFile
-                                                        represents information to
-                                                        create the file containing
-                                                        the pod field
-                                                      type: object
-                                                      required:
-                                                      - path
-                                                      properties:
-                                                        fieldRef:
-                                                          description: 'Required:
-                                                            Selects a field of the
-                                                            pod: only annotations,
-                                                            labels, name and namespace
-                                                            are supported.'
-                                                          type: object
-                                                          required:
-                                                          - fieldPath
-                                                          properties:
-                                                            apiVersion:
-                                                              description: Version
-                                                                of the schema the
-                                                                FieldPath is written
-                                                                in terms of, defaults
-                                                                to "v1".
-                                                              type: string
-                                                            fieldPath:
-                                                              description: Path of
-                                                                the field to select
-                                                                in the specified API
-                                                                version.
-                                                              type: string
-                                                        mode:
-                                                          description: 'Optional:
-                                                            mode bits to use on this
-                                                            file, must be a value
-                                                            between 0 and 0777. If
-                                                            not specified, the volume
-                                                            defaultMode will be used.
-                                                            This might be in conflict
-                                                            with other options that
-                                                            affect the file mode,
-                                                            like fsGroup, and the
-                                                            result can be other mode
-                                                            bits set.'
-                                                          type: integer
-                                                          format: int32
-                                                        path:
-                                                          description: 'Required:
-                                                            Path is  the relative
-                                                            path name of the file
-                                                            to be created. Must not
-                                                            be absolute or contain
-                                                            the ''..'' path. Must
-                                                            be utf-8 encoded. The
-                                                            first item of the relative
-                                                            path must not start with
-                                                            ''..'''
-                                                          type: string
-                                                        resourceFieldRef:
-                                                          description: 'Selects a
-                                                            resource of the container:
-                                                            only resources limits
-                                                            and requests (limits.cpu,
-                                                            limits.memory, requests.cpu
-                                                            and requests.memory) are
-                                                            currently supported.'
-                                                          type: object
-                                                          required:
-                                                          - resource
-                                                          properties:
-                                                            containerName:
-                                                              description: 'Container
-                                                                name: required for
-                                                                volumes, optional
-                                                                for env vars'
-                                                              type: string
-                                                            divisor:
-                                                              description: Specifies
-                                                                the output format
-                                                                of the exposed resources,
-                                                                defaults to "1"
-                                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                              anyOf:
-                                                              - type: integer
-                                                              - type: string
-                                                              x-kubernetes-int-or-string: true
-                                                            resource:
-                                                              description: 'Required:
-                                                                resource to select'
-                                                              type: string
-                                              emptyDir:
-                                                description: 'EmptyDir represents
-                                                  a temporary directory that shares
-                                                  a pod''s lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir'
-                                                type: object
-                                                properties:
-                                                  medium:
-                                                    description: 'What type of storage
-                                                      medium should back this directory.
-                                                      The default is "" which means
-                                                      to use the node''s default medium.
-                                                      Must be an empty string (default)
-                                                      or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir'
-                                                    type: string
-                                                  sizeLimit:
-                                                    description: 'Total amount of
-                                                      local storage required for this
-                                                      EmptyDir volume. The size limit
-                                                      is also applicable for memory
-                                                      medium. The maximum usage on
-                                                      memory medium EmptyDir would
-                                                      be the minimum value between
-                                                      the SizeLimit specified here
-                                                      and the sum of memory limits
-                                                      of all containers in a pod.
-                                                      The default is nil which means
-                                                      that the limit is undefined.
-                                                      More info: http://kubernetes.io/docs/user-guide/volumes#emptydir'
-                                                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                    anyOf:
-                                                    - type: integer
-                                                    - type: string
-                                                    x-kubernetes-int-or-string: true
-                                              fc:
-                                                description: FC represents a Fibre
-                                                  Channel resource that is attached
-                                                  to a kubelet's host machine and
-                                                  then exposed to the pod.
-                                                type: object
-                                                properties:
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      to mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified. TODO: how do
-                                                      we prevent errors in the filesystem
-                                                      from compromising the machine'
-                                                    type: string
-                                                  lun:
-                                                    description: 'Optional: FC target
-                                                      lun number'
-                                                    type: integer
-                                                    format: int32
-                                                  readOnly:
-                                                    description: 'Optional: Defaults
-                                                      to false (read/write). ReadOnly
-                                                      here will force the ReadOnly
-                                                      setting in VolumeMounts.'
-                                                    type: boolean
-                                                  targetWWNs:
-                                                    description: 'Optional: FC target
-                                                      worldwide names (WWNs)'
-                                                    type: array
-                                                    items:
-                                                      type: string
-                                                  wwids:
-                                                    description: 'Optional: FC volume
-                                                      world wide identifiers (wwids)
-                                                      Either wwids or combination
-                                                      of targetWWNs and lun must be
-                                                      set, but not both simultaneously.'
-                                                    type: array
-                                                    items:
-                                                      type: string
-                                              flexVolume:
-                                                description: FlexVolume represents
-                                                  a generic volume resource that is
-                                                  provisioned/attached using an exec
-                                                  based plugin.
-                                                type: object
-                                                required:
-                                                - driver
-                                                properties:
-                                                  driver:
-                                                    description: Driver is the name
-                                                      of the driver to use for this
-                                                      volume.
-                                                    type: string
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      The default filesystem depends
-                                                      on FlexVolume script.
-                                                    type: string
-                                                  options:
-                                                    description: 'Optional: Extra
-                                                      command options if any.'
-                                                    type: object
-                                                    additionalProperties:
-                                                      type: string
-                                                  readOnly:
-                                                    description: 'Optional: Defaults
-                                                      to false (read/write). ReadOnly
-                                                      here will force the ReadOnly
-                                                      setting in VolumeMounts.'
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: 'Optional: SecretRef
-                                                      is reference to the secret object
-                                                      containing sensitive information
-                                                      to pass to the plugin scripts.
-                                                      This may be empty if no secret
-                                                      object is specified. If the
-                                                      secret object contains more
-                                                      than one secret, all secrets
-                                                      are passed to the plugin scripts.'
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                              flocker:
-                                                description: Flocker represents a
-                                                  Flocker volume attached to a kubelet's
-                                                  host machine. This depends on the
-                                                  Flocker control service being running
-                                                type: object
-                                                properties:
-                                                  datasetName:
-                                                    description: Name of the dataset
-                                                      stored as metadata -> name on
-                                                      the dataset for Flocker should
-                                                      be considered as deprecated
-                                                    type: string
-                                                  datasetUUID:
-                                                    description: UUID of the dataset.
-                                                      This is unique identifier of
-                                                      a Flocker dataset
-                                                    type: string
-                                              gcePersistentDisk:
-                                                description: 'GCEPersistentDisk represents
-                                                  a GCE Disk resource that is attached
-                                                  to a kubelet''s host machine and
-                                                  then exposed to the pod. More info:
-                                                  https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                                                type: object
-                                                required:
-                                                - pdName
-                                                properties:
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      of the volume that you want
-                                                      to mount. Tip: Ensure that the
-                                                      filesystem type is supported
-                                                      by the host operating system.
-                                                      Examples: "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
-                                                      TODO: how do we prevent errors
-                                                      in the filesystem from compromising
-                                                      the machine'
-                                                    type: string
-                                                  partition:
-                                                    description: 'The partition in
-                                                      the volume that you want to
-                                                      mount. If omitted, the default
-                                                      is to mount by volume name.
-                                                      Examples: For volume /dev/sda1,
-                                                      you specify the partition as
-                                                      "1". Similarly, the volume partition
-                                                      for /dev/sda is "0" (or you
-                                                      can leave the property empty).
-                                                      More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                                                    type: integer
-                                                    format: int32
-                                                  pdName:
-                                                    description: 'Unique name of the
-                                                      PD resource in GCE. Used to
-                                                      identify the disk in GCE. More
-                                                      info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'ReadOnly here will
-                                                      force the ReadOnly setting in
-                                                      VolumeMounts. Defaults to false.
-                                                      More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                                                    type: boolean
-                                              gitRepo:
-                                                description: 'GitRepo represents a
-                                                  git repository at a particular revision.
-                                                  DEPRECATED: GitRepo is deprecated.
-                                                  To provision a container with a
-                                                  git repo, mount an EmptyDir into
-                                                  an InitContainer that clones the
-                                                  repo using git, then mount the EmptyDir
-                                                  into the Pod''s container.'
-                                                type: object
-                                                required:
-                                                - repository
-                                                properties:
-                                                  directory:
-                                                    description: Target directory
-                                                      name. Must not contain or start
-                                                      with '..'.  If '.' is supplied,
-                                                      the volume directory will be
-                                                      the git repository.  Otherwise,
-                                                      if specified, the volume will
-                                                      contain the git repository in
-                                                      the subdirectory with the given
-                                                      name.
-                                                    type: string
-                                                  repository:
-                                                    description: Repository URL
-                                                    type: string
-                                                  revision:
-                                                    description: Commit hash for the
-                                                      specified revision.
-                                                    type: string
-                                              glusterfs:
-                                                description: 'Glusterfs represents
-                                                  a Glusterfs mount on the host that
-                                                  shares a pod''s lifetime. More info:
-                                                  https://examples.k8s.io/volumes/glusterfs/README.md'
-                                                type: object
-                                                required:
-                                                - endpoints
-                                                - path
-                                                properties:
-                                                  endpoints:
-                                                    description: 'EndpointsName is
-                                                      the endpoint name that details
-                                                      Glusterfs topology. More info:
-                                                      https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                                                    type: string
-                                                  path:
-                                                    description: 'Path is the Glusterfs
-                                                      volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'ReadOnly here will
-                                                      force the Glusterfs volume to
-                                                      be mounted with read-only permissions.
-                                                      Defaults to false. More info:
-                                                      https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                                                    type: boolean
-                                              hostPath:
-                                                description: 'HostPath represents
-                                                  a pre-existing file or directory
-                                                  on the host machine that is directly
-                                                  exposed to the container. This is
-                                                  generally used for system agents
-                                                  or other privileged things that
-                                                  are allowed to see the host machine.
-                                                  Most containers will NOT need this.
-                                                  More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
-                                                  --- TODO(jonesdl) We need to restrict
-                                                  who can use host directory mounts
-                                                  and who can/can not mount host directories
-                                                  as read/write.'
-                                                type: object
-                                                required:
-                                                - path
-                                                properties:
-                                                  path:
-                                                    description: 'Path of the directory
-                                                      on the host. If the path is
-                                                      a symlink, it will follow the
-                                                      link to the real path. More
-                                                      info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath'
-                                                    type: string
-                                                  type:
-                                                    description: 'Type for HostPath
-                                                      Volume Defaults to "" More info:
-                                                      https://kubernetes.io/docs/concepts/storage/volumes#hostpath'
-                                                    type: string
-                                              iscsi:
-                                                description: 'ISCSI represents an
-                                                  ISCSI Disk resource that is attached
-                                                  to a kubelet''s host machine and
-                                                  then exposed to the pod. More info:
-                                                  https://examples.k8s.io/volumes/iscsi/README.md'
-                                                type: object
-                                                required:
-                                                - iqn
-                                                - lun
-                                                - targetPortal
-                                                properties:
-                                                  chapAuthDiscovery:
-                                                    description: whether support iSCSI
-                                                      Discovery CHAP authentication
-                                                    type: boolean
-                                                  chapAuthSession:
-                                                    description: whether support iSCSI
-                                                      Session CHAP authentication
-                                                    type: boolean
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      of the volume that you want
-                                                      to mount. Tip: Ensure that the
-                                                      filesystem type is supported
-                                                      by the host operating system.
-                                                      Examples: "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi
-                                                      TODO: how do we prevent errors
-                                                      in the filesystem from compromising
-                                                      the machine'
-                                                    type: string
-                                                  initiatorName:
-                                                    description: Custom iSCSI Initiator
-                                                      Name. If initiatorName is specified
-                                                      with iscsiInterface simultaneously,
-                                                      new iSCSI interface <target
-                                                      portal>:<volume name> will be
-                                                      created for the connection.
-                                                    type: string
-                                                  iqn:
-                                                    description: Target iSCSI Qualified
-                                                      Name.
-                                                    type: string
-                                                  iscsiInterface:
-                                                    description: iSCSI Interface Name
-                                                      that uses an iSCSI transport.
-                                                      Defaults to 'default' (tcp).
-                                                    type: string
-                                                  lun:
-                                                    description: iSCSI Target Lun
-                                                      number.
-                                                    type: integer
-                                                    format: int32
-                                                  portals:
-                                                    description: iSCSI Target Portal
-                                                      List. The portal is either an
-                                                      IP or ip_addr:port if the port
-                                                      is other than default (typically
-                                                      TCP ports 860 and 3260).
-                                                    type: array
-                                                    items:
-                                                      type: string
-                                                  readOnly:
-                                                    description: ReadOnly here will
-                                                      force the ReadOnly setting in
-                                                      VolumeMounts. Defaults to false.
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: CHAP Secret for iSCSI
-                                                      target and initiator authentication
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  targetPortal:
-                                                    description: iSCSI Target Portal.
-                                                      The Portal is either an IP or
-                                                      ip_addr:port if the port is
-                                                      other than default (typically
-                                                      TCP ports 860 and 3260).
-                                                    type: string
-                                              name:
-                                                description: 'Volume''s name. Must
-                                                  be a DNS_LABEL and unique within
-                                                  the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                                                type: string
-                                              nfs:
-                                                description: 'NFS represents an NFS
-                                                  mount on the host that shares a
-                                                  pod''s lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                                                type: object
-                                                required:
-                                                - path
-                                                - server
-                                                properties:
-                                                  path:
-                                                    description: 'Path that is exported
-                                                      by the NFS server. More info:
-                                                      https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'ReadOnly here will
-                                                      force the NFS export to be mounted
-                                                      with read-only permissions.
-                                                      Defaults to false. More info:
-                                                      https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                                                    type: boolean
-                                                  server:
-                                                    description: 'Server is the hostname
-                                                      or IP address of the NFS server.
-                                                      More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                                                    type: string
-                                              persistentVolumeClaim:
-                                                description: 'PersistentVolumeClaimVolumeSource
-                                                  represents a reference to a PersistentVolumeClaim
-                                                  in the same namespace. More info:
-                                                  https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
-                                                type: object
-                                                required:
-                                                - claimName
-                                                properties:
-                                                  claimName:
-                                                    description: 'ClaimName is the
-                                                      name of a PersistentVolumeClaim
-                                                      in the same namespace as the
-                                                      pod using this volume. More
-                                                      info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
-                                                    type: string
-                                                  readOnly:
-                                                    description: Will force the ReadOnly
-                                                      setting in VolumeMounts. Default
-                                                      false.
-                                                    type: boolean
-                                              photonPersistentDisk:
-                                                description: PhotonPersistentDisk
-                                                  represents a PhotonController persistent
-                                                  disk attached and mounted on kubelets
-                                                  host machine
-                                                type: object
-                                                required:
-                                                - pdID
-                                                properties:
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified.
-                                                    type: string
-                                                  pdID:
-                                                    description: ID that identifies
-                                                      Photon Controller persistent
-                                                      disk
-                                                    type: string
-                                              portworxVolume:
-                                                description: PortworxVolume represents
-                                                  a portworx volume attached and mounted
-                                                  on kubelets host machine
-                                                type: object
-                                                required:
-                                                - volumeID
-                                                properties:
-                                                  fsType:
-                                                    description: FSType represents
-                                                      the filesystem type to mount
-                                                      Must be a filesystem type supported
-                                                      by the host operating system.
-                                                      Ex. "ext4", "xfs". Implicitly
-                                                      inferred to be "ext4" if unspecified.
-                                                    type: string
-                                                  readOnly:
-                                                    description: Defaults to false
-                                                      (read/write). ReadOnly here
-                                                      will force the ReadOnly setting
-                                                      in VolumeMounts.
-                                                    type: boolean
-                                                  volumeID:
-                                                    description: VolumeID uniquely
-                                                      identifies a Portworx volume
-                                                    type: string
-                                              projected:
-                                                description: Items for all in one
-                                                  resources secrets, configmaps, and
-                                                  downward API
-                                                type: object
-                                                required:
-                                                - sources
-                                                properties:
-                                                  defaultMode:
-                                                    description: Mode bits to use
-                                                      on created files by default.
-                                                      Must be a value between 0 and
-                                                      0777. Directories within the
-                                                      path are not affected by this
-                                                      setting. This might be in conflict
-                                                      with other options that affect
-                                                      the file mode, like fsGroup,
-                                                      and the result can be other
-                                                      mode bits set.
-                                                    type: integer
-                                                    format: int32
-                                                  sources:
-                                                    description: list of volume projections
-                                                    type: array
-                                                    items:
-                                                      description: Projection that
-                                                        may be projected along with
-                                                        other supported volume types
-                                                      type: object
-                                                      properties:
-                                                        configMap:
-                                                          description: information
-                                                            about the configMap data
-                                                            to project
-                                                          type: object
-                                                          properties:
-                                                            items:
-                                                              description: If unspecified,
-                                                                each key-value pair
-                                                                in the Data field
-                                                                of the referenced
-                                                                ConfigMap will be
-                                                                projected into the
-                                                                volume as a file whose
-                                                                name is the key and
-                                                                content is the value.
-                                                                If specified, the
-                                                                listed keys will be
-                                                                projected into the
-                                                                specified paths, and
-                                                                unlisted keys will
-                                                                not be present. If
-                                                                a key is specified
-                                                                which is not present
-                                                                in the ConfigMap,
-                                                                the volume setup will
-                                                                error unless it is
-                                                                marked optional. Paths
-                                                                must be relative and
-                                                                may not contain the
-                                                                '..' path or start
-                                                                with '..'.
-                                                              type: array
-                                                              items:
-                                                                description: Maps
-                                                                  a string key to
-                                                                  a path within a
-                                                                  volume.
-                                                                type: object
-                                                                required:
-                                                                - key
-                                                                - path
-                                                                properties:
-                                                                  key:
-                                                                    description: The
-                                                                      key to project.
-                                                                    type: string
-                                                                  mode:
-                                                                    description: 'Optional:
-                                                                      mode bits to
-                                                                      use on this
-                                                                      file, must be
-                                                                      a value between
-                                                                      0 and 0777.
-                                                                      If not specified,
-                                                                      the volume defaultMode
-                                                                      will be used.
-                                                                      This might be
-                                                                      in conflict
-                                                                      with other options
-                                                                      that affect
-                                                                      the file mode,
-                                                                      like fsGroup,
-                                                                      and the result
-                                                                      can be other
-                                                                      mode bits set.'
-                                                                    type: integer
-                                                                    format: int32
-                                                                  path:
-                                                                    description: The
-                                                                      relative path
-                                                                      of the file
-                                                                      to map the key
-                                                                      to. May not
-                                                                      be an absolute
-                                                                      path. May not
-                                                                      contain the
-                                                                      path element
-                                                                      '..'. May not
-                                                                      start with the
-                                                                      string '..'.
-                                                                    type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the ConfigMap
-                                                                or its keys must be
-                                                                defined
-                                                              type: boolean
-                                                        downwardAPI:
-                                                          description: information
-                                                            about the downwardAPI
-                                                            data to project
-                                                          type: object
-                                                          properties:
-                                                            items:
-                                                              description: Items is
-                                                                a list of DownwardAPIVolume
-                                                                file
-                                                              type: array
-                                                              items:
-                                                                description: DownwardAPIVolumeFile
-                                                                  represents information
-                                                                  to create the file
-                                                                  containing the pod
-                                                                  field
-                                                                type: object
-                                                                required:
-                                                                - path
-                                                                properties:
-                                                                  fieldRef:
-                                                                    description: 'Required:
-                                                                      Selects a field
-                                                                      of the pod:
-                                                                      only annotations,
-                                                                      labels, name
-                                                                      and namespace
-                                                                      are supported.'
-                                                                    type: object
-                                                                    required:
-                                                                    - fieldPath
-                                                                    properties:
-                                                                      apiVersion:
-                                                                        description: Version
-                                                                          of the schema
-                                                                          the FieldPath
-                                                                          is written
-                                                                          in terms
-                                                                          of, defaults
-                                                                          to "v1".
-                                                                        type: string
-                                                                      fieldPath:
-                                                                        description: Path
-                                                                          of the field
-                                                                          to select
-                                                                          in the specified
-                                                                          API version.
-                                                                        type: string
-                                                                  mode:
-                                                                    description: 'Optional:
-                                                                      mode bits to
-                                                                      use on this
-                                                                      file, must be
-                                                                      a value between
-                                                                      0 and 0777.
-                                                                      If not specified,
-                                                                      the volume defaultMode
-                                                                      will be used.
-                                                                      This might be
-                                                                      in conflict
-                                                                      with other options
-                                                                      that affect
-                                                                      the file mode,
-                                                                      like fsGroup,
-                                                                      and the result
-                                                                      can be other
-                                                                      mode bits set.'
-                                                                    type: integer
-                                                                    format: int32
-                                                                  path:
-                                                                    description: 'Required:
-                                                                      Path is  the
-                                                                      relative path
-                                                                      name of the
-                                                                      file to be created.
-                                                                      Must not be
-                                                                      absolute or
-                                                                      contain the
-                                                                      ''..'' path.
-                                                                      Must be utf-8
-                                                                      encoded. The
-                                                                      first item of
-                                                                      the relative
-                                                                      path must not
-                                                                      start with ''..'''
-                                                                    type: string
-                                                                  resourceFieldRef:
-                                                                    description: 'Selects
-                                                                      a resource of
-                                                                      the container:
-                                                                      only resources
-                                                                      limits and requests
-                                                                      (limits.cpu,
-                                                                      limits.memory,
-                                                                      requests.cpu
-                                                                      and requests.memory)
-                                                                      are currently
-                                                                      supported.'
-                                                                    type: object
-                                                                    required:
-                                                                    - resource
-                                                                    properties:
-                                                                      containerName:
-                                                                        description: 'Container
-                                                                          name: required
-                                                                          for volumes,
-                                                                          optional
-                                                                          for env
-                                                                          vars'
-                                                                        type: string
-                                                                      divisor:
-                                                                        description: Specifies
-                                                                          the output
-                                                                          format of
-                                                                          the exposed
-                                                                          resources,
-                                                                          defaults
-                                                                          to "1"
-                                                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                                        anyOf:
-                                                                        - type: integer
-                                                                        - type: string
-                                                                        x-kubernetes-int-or-string: true
-                                                                      resource:
-                                                                        description: 'Required:
-                                                                          resource
-                                                                          to select'
-                                                                        type: string
-                                                        secret:
-                                                          description: information
-                                                            about the secret data
-                                                            to project
-                                                          type: object
-                                                          properties:
-                                                            items:
-                                                              description: If unspecified,
-                                                                each key-value pair
-                                                                in the Data field
-                                                                of the referenced
-                                                                Secret will be projected
-                                                                into the volume as
-                                                                a file whose name
-                                                                is the key and content
-                                                                is the value. If specified,
-                                                                the listed keys will
-                                                                be projected into
-                                                                the specified paths,
-                                                                and unlisted keys
-                                                                will not be present.
-                                                                If a key is specified
-                                                                which is not present
-                                                                in the Secret, the
-                                                                volume setup will
-                                                                error unless it is
-                                                                marked optional. Paths
-                                                                must be relative and
-                                                                may not contain the
-                                                                '..' path or start
-                                                                with '..'.
-                                                              type: array
-                                                              items:
-                                                                description: Maps
-                                                                  a string key to
-                                                                  a path within a
-                                                                  volume.
-                                                                type: object
-                                                                required:
-                                                                - key
-                                                                - path
-                                                                properties:
-                                                                  key:
-                                                                    description: The
-                                                                      key to project.
-                                                                    type: string
-                                                                  mode:
-                                                                    description: 'Optional:
-                                                                      mode bits to
-                                                                      use on this
-                                                                      file, must be
-                                                                      a value between
-                                                                      0 and 0777.
-                                                                      If not specified,
-                                                                      the volume defaultMode
-                                                                      will be used.
-                                                                      This might be
-                                                                      in conflict
-                                                                      with other options
-                                                                      that affect
-                                                                      the file mode,
-                                                                      like fsGroup,
-                                                                      and the result
-                                                                      can be other
-                                                                      mode bits set.'
-                                                                    type: integer
-                                                                    format: int32
-                                                                  path:
-                                                                    description: The
-                                                                      relative path
-                                                                      of the file
-                                                                      to map the key
-                                                                      to. May not
-                                                                      be an absolute
-                                                                      path. May not
-                                                                      contain the
-                                                                      path element
-                                                                      '..'. May not
-                                                                      start with the
-                                                                      string '..'.
-                                                                    type: string
-                                                            name:
-                                                              description: 'Name of
-                                                                the referent. More
-                                                                info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                                TODO: Add other useful
-                                                                fields. apiVersion,
-                                                                kind, uid?'
-                                                              type: string
-                                                            optional:
-                                                              description: Specify
-                                                                whether the Secret
-                                                                or its key must be
-                                                                defined
-                                                              type: boolean
-                                                        serviceAccountToken:
-                                                          description: information
-                                                            about the serviceAccountToken
-                                                            data to project
-                                                          type: object
-                                                          required:
-                                                          - path
-                                                          properties:
-                                                            audience:
-                                                              description: Audience
-                                                                is the intended audience
-                                                                of the token. A recipient
-                                                                of a token must identify
-                                                                itself with an identifier
-                                                                specified in the audience
-                                                                of the token, and
-                                                                otherwise should reject
-                                                                the token. The audience
-                                                                defaults to the identifier
-                                                                of the apiserver.
-                                                              type: string
-                                                            expirationSeconds:
-                                                              description: ExpirationSeconds
-                                                                is the requested duration
-                                                                of validity of the
-                                                                service account token.
-                                                                As the token approaches
-                                                                expiration, the kubelet
-                                                                volume plugin will
-                                                                proactively rotate
-                                                                the service account
-                                                                token. The kubelet
-                                                                will start trying
-                                                                to rotate the token
-                                                                if the token is older
-                                                                than 80 percent of
-                                                                its time to live or
-                                                                if the token is older
-                                                                than 24 hours.Defaults
-                                                                to 1 hour and must
-                                                                be at least 10 minutes.
-                                                              type: integer
-                                                              format: int64
-                                                            path:
-                                                              description: Path is
-                                                                the path relative
-                                                                to the mount point
-                                                                of the file to project
-                                                                the token into.
-                                                              type: string
-                                              quobyte:
-                                                description: Quobyte represents a
-                                                  Quobyte mount on the host that shares
-                                                  a pod's lifetime
-                                                type: object
-                                                required:
-                                                - registry
-                                                - volume
-                                                properties:
-                                                  group:
-                                                    description: Group to map volume
-                                                      access to Default is no group
-                                                    type: string
-                                                  readOnly:
-                                                    description: ReadOnly here will
-                                                      force the Quobyte volume to
-                                                      be mounted with read-only permissions.
-                                                      Defaults to false.
-                                                    type: boolean
-                                                  registry:
-                                                    description: Registry represents
-                                                      a single or multiple Quobyte
-                                                      Registry services specified
-                                                      as a string as host:port pair
-                                                      (multiple entries are separated
-                                                      with commas) which acts as the
-                                                      central registry for volumes
-                                                    type: string
-                                                  tenant:
-                                                    description: Tenant owning the
-                                                      given Quobyte volume in the
-                                                      Backend Used with dynamically
-                                                      provisioned Quobyte volumes,
-                                                      value is set by the plugin
-                                                    type: string
-                                                  user:
-                                                    description: User to map volume
-                                                      access to Defaults to serivceaccount
-                                                      user
-                                                    type: string
-                                                  volume:
-                                                    description: Volume is a string
-                                                      that references an already created
-                                                      Quobyte volume by name.
-                                                    type: string
-                                              rbd:
-                                                description: 'RBD represents a Rados
-                                                  Block Device mount on the host that
-                                                  shares a pod''s lifetime. More info:
-                                                  https://examples.k8s.io/volumes/rbd/README.md'
-                                                type: object
-                                                required:
-                                                - image
-                                                - monitors
-                                                properties:
-                                                  fsType:
-                                                    description: 'Filesystem type
-                                                      of the volume that you want
-                                                      to mount. Tip: Ensure that the
-                                                      filesystem type is supported
-                                                      by the host operating system.
-                                                      Examples: "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd
-                                                      TODO: how do we prevent errors
-                                                      in the filesystem from compromising
-                                                      the machine'
-                                                    type: string
-                                                  image:
-                                                    description: 'The rados image
-                                                      name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: string
-                                                  keyring:
-                                                    description: 'Keyring is the path
-                                                      to key ring for RBDUser. Default
-                                                      is /etc/ceph/keyring. More info:
-                                                      https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: string
-                                                  monitors:
-                                                    description: 'A collection of
-                                                      Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: array
-                                                    items:
-                                                      type: string
-                                                  pool:
-                                                    description: 'The rados pool name.
-                                                      Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: string
-                                                  readOnly:
-                                                    description: 'ReadOnly here will
-                                                      force the ReadOnly setting in
-                                                      VolumeMounts. Defaults to false.
-                                                      More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: 'SecretRef is name
-                                                      of the authentication secret
-                                                      for RBDUser. If provided overrides
-                                                      keyring. Default is nil. More
-                                                      info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  user:
-                                                    description: 'The rados user name.
-                                                      Default is admin. More info:
-                                                      https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                                                    type: string
-                                              scaleIO:
-                                                description: ScaleIO represents a
-                                                  ScaleIO persistent volume attached
-                                                  and mounted on Kubernetes nodes.
-                                                type: object
-                                                required:
-                                                - gateway
-                                                - secretRef
-                                                - system
-                                                properties:
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Default is "xfs".
-                                                    type: string
-                                                  gateway:
-                                                    description: The host address
-                                                      of the ScaleIO API Gateway.
-                                                    type: string
-                                                  protectionDomain:
-                                                    description: The name of the ScaleIO
-                                                      Protection Domain for the configured
-                                                      storage.
-                                                    type: string
-                                                  readOnly:
-                                                    description: Defaults to false
-                                                      (read/write). ReadOnly here
-                                                      will force the ReadOnly setting
-                                                      in VolumeMounts.
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: SecretRef references
-                                                      to the secret for ScaleIO user
-                                                      and other sensitive information.
-                                                      If this is not provided, Login
-                                                      operation will fail.
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  sslEnabled:
-                                                    description: Flag to enable/disable
-                                                      SSL communication with Gateway,
-                                                      default false
-                                                    type: boolean
-                                                  storageMode:
-                                                    description: Indicates whether
-                                                      the storage for a volume should
-                                                      be ThickProvisioned or ThinProvisioned.
-                                                      Default is ThinProvisioned.
-                                                    type: string
-                                                  storagePool:
-                                                    description: The ScaleIO Storage
-                                                      Pool associated with the protection
-                                                      domain.
-                                                    type: string
-                                                  system:
-                                                    description: The name of the storage
-                                                      system as configured in ScaleIO.
-                                                    type: string
-                                                  volumeName:
-                                                    description: The name of a volume
-                                                      already created in the ScaleIO
-                                                      system that is associated with
-                                                      this volume source.
-                                                    type: string
-                                              secret:
-                                                description: 'Secret represents a
-                                                  secret that should populate this
-                                                  volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret'
-                                                type: object
-                                                properties:
-                                                  defaultMode:
-                                                    description: 'Optional: mode bits
-                                                      to use on created files by default.
-                                                      Must be a value between 0 and
-                                                      0777. Defaults to 0644. Directories
-                                                      within the path are not affected
-                                                      by this setting. This might
-                                                      be in conflict with other options
-                                                      that affect the file mode, like
-                                                      fsGroup, and the result can
-                                                      be other mode bits set.'
-                                                    type: integer
-                                                    format: int32
-                                                  items:
-                                                    description: If unspecified, each
-                                                      key-value pair in the Data field
-                                                      of the referenced Secret will
-                                                      be projected into the volume
-                                                      as a file whose name is the
-                                                      key and content is the value.
-                                                      If specified, the listed keys
-                                                      will be projected into the specified
-                                                      paths, and unlisted keys will
-                                                      not be present. If a key is
-                                                      specified which is not present
-                                                      in the Secret, the volume setup
-                                                      will error unless it is marked
-                                                      optional. Paths must be relative
-                                                      and may not contain the '..'
-                                                      path or start with '..'.
-                                                    type: array
-                                                    items:
-                                                      description: Maps a string key
-                                                        to a path within a volume.
-                                                      type: object
-                                                      required:
-                                                      - key
-                                                      - path
-                                                      properties:
-                                                        key:
-                                                          description: The key to
-                                                            project.
-                                                          type: string
-                                                        mode:
-                                                          description: 'Optional:
-                                                            mode bits to use on this
-                                                            file, must be a value
-                                                            between 0 and 0777. If
-                                                            not specified, the volume
-                                                            defaultMode will be used.
-                                                            This might be in conflict
-                                                            with other options that
-                                                            affect the file mode,
-                                                            like fsGroup, and the
-                                                            result can be other mode
-                                                            bits set.'
-                                                          type: integer
-                                                          format: int32
-                                                        path:
-                                                          description: The relative
-                                                            path of the file to map
-                                                            the key to. May not be
-                                                            an absolute path. May
-                                                            not contain the path element
-                                                            '..'. May not start with
-                                                            the string '..'.
-                                                          type: string
-                                                  optional:
-                                                    description: Specify whether the
-                                                      Secret or its keys must be defined
-                                                    type: boolean
-                                                  secretName:
-                                                    description: 'Name of the secret
-                                                      in the pod''s namespace to use.
-                                                      More info: https://kubernetes.io/docs/concepts/storage/volumes#secret'
-                                                    type: string
-                                              storageos:
-                                                description: StorageOS represents
-                                                  a StorageOS volume attached and
-                                                  mounted on Kubernetes nodes.
-                                                type: object
-                                                properties:
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified.
-                                                    type: string
-                                                  readOnly:
-                                                    description: Defaults to false
-                                                      (read/write). ReadOnly here
-                                                      will force the ReadOnly setting
-                                                      in VolumeMounts.
-                                                    type: boolean
-                                                  secretRef:
-                                                    description: SecretRef specifies
-                                                      the secret to use for obtaining
-                                                      the StorageOS API credentials.  If
-                                                      not specified, default values
-                                                      will be attempted.
-                                                    type: object
-                                                    properties:
-                                                      name:
-                                                        description: 'Name of the
-                                                          referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                                          TODO: Add other useful fields.
-                                                          apiVersion, kind, uid?'
-                                                        type: string
-                                                  volumeName:
-                                                    description: VolumeName is the
-                                                      human-readable name of the StorageOS
-                                                      volume.  Volume names are only
-                                                      unique within a namespace.
-                                                    type: string
-                                                  volumeNamespace:
-                                                    description: VolumeNamespace specifies
-                                                      the scope of the volume within
-                                                      StorageOS.  If no namespace
-                                                      is specified then the Pod's
-                                                      namespace will be used.  This
-                                                      allows the Kubernetes name scoping
-                                                      to be mirrored within StorageOS
-                                                      for tighter integration. Set
-                                                      VolumeName to any name to override
-                                                      the default behaviour. Set to
-                                                      "default" if you are not using
-                                                      namespaces within StorageOS.
-                                                      Namespaces that do not pre-exist
-                                                      within StorageOS will be created.
-                                                    type: string
-                                              vsphereVolume:
-                                                description: VsphereVolume represents
-                                                  a vSphere volume attached and mounted
-                                                  on kubelets host machine
-                                                type: object
-                                                required:
-                                                - volumePath
-                                                properties:
-                                                  fsType:
-                                                    description: Filesystem type to
-                                                      mount. Must be a filesystem
-                                                      type supported by the host operating
-                                                      system. Ex. "ext4", "xfs", "ntfs".
-                                                      Implicitly inferred to be "ext4"
-                                                      if unspecified.
-                                                    type: string
-                                                  storagePolicyID:
-                                                    description: Storage Policy Based
-                                                      Management (SPBM) profile ID
-                                                      associated with the StoragePolicyName.
-                                                    type: string
-                                                  storagePolicyName:
-                                                    description: Storage Policy Based
-                                                      Management (SPBM) profile name.
-                                                    type: string
-                                                  volumePath:
-                                                    description: Path that identifies
-                                                      vSphere volume vmdk
-                                                    type: string
-                      permissions:
-                        type: array
-                        items:
-                          description: StrategyDeploymentPermissions describe the
-                            rbac rules and service account needed by the install strategy
-                          type: object
-                          required:
-                          - rules
-                          - serviceAccountName
-                          properties:
-                            rules:
-                              type: array
-                              items:
-                                description: PolicyRule holds information that describes
-                                  a policy rule, but does not contain information
-                                  about who the rule applies to or which namespace
-                                  the rule applies to.
-                                type: object
-                                required:
-                                - verbs
-                                properties:
-                                  apiGroups:
-                                    description: APIGroups is the name of the APIGroup
-                                      that contains the resources.  If multiple API
-                                      groups are specified, any action requested against
-                                      one of the enumerated resources in any API group
-                                      will be allowed.
-                                    type: array
-                                    items:
-                                      type: string
-                                  nonResourceURLs:
-                                    description: NonResourceURLs is a set of partial
-                                      urls that a user should have access to.  *s
-                                      are allowed, but only as the full, final step
-                                      in the path Since non-resource URLs are not
-                                      namespaced, this field is only applicable for
-                                      ClusterRoles referenced from a ClusterRoleBinding.
-                                      Rules can either apply to API resources (such
-                                      as "pods" or "secrets") or non-resource URL
-                                      paths (such as "/api"),  but not both.
-                                    type: array
-                                    items:
-                                      type: string
-                                  resourceNames:
-                                    description: ResourceNames is an optional white
-                                      list of names that the rule applies to.  An
-                                      empty set means that everything is allowed.
-                                    type: array
-                                    items:
-                                      type: string
-                                  resources:
-                                    description: Resources is a list of resources
-                                      this rule applies to.  ResourceAll represents
-                                      all resources.
-                                    type: array
-                                    items:
-                                      type: string
-                                  verbs:
-                                    description: Verbs is a list of Verbs that apply
-                                      to ALL the ResourceKinds and AttributeRestrictions
-                                      contained in this rule.  VerbAll represents
-                                      all kinds.
-                                    type: array
-                                    items:
-                                      type: string
-                            serviceAccountName:
-                              type: string
-                  strategy:
-                    type: string
-              installModes:
-                description: InstallModes specify supported installation types
-                type: array
-                items:
-                  description: InstallMode associates an InstallModeType with a flag
-                    representing if the CSV supports it
-                  type: object
-                  required:
-                  - supported
-                  - type
-                  properties:
-                    supported:
-                      type: boolean
-                    type:
-                      description: InstallModeType is a supported type of install
-                        mode for CSV installation
-                      type: string
-              keywords:
-                type: array
-                items:
-                  type: string
-              labels:
-                description: Map of string keys and values that can be used to organize
-                  and categorize (scope and select) objects.
-                type: object
-                additionalProperties:
-                  type: string
-              links:
-                type: array
-                items:
-                  type: object
-                  properties:
-                    name:
-                      type: string
-                    url:
-                      type: string
-              maintainers:
-                type: array
-                items:
-                  type: object
-                  properties:
-                    email:
-                      type: string
-                    name:
-                      type: string
-              maturity:
-                type: string
-              minKubeVersion:
-                type: string
-              nativeAPIs:
-                type: array
-                items:
-                  description: GroupVersionKind unambiguously identifies a kind.  It
-                    doesn't anonymously include GroupVersion to avoid automatic coersion.  It
-                    doesn't use a GroupVersion to avoid custom marshalling
-                  type: object
-                  required:
-                  - group
-                  - kind
-                  - version
-                  properties:
-                    group:
-                      type: string
-                    kind:
-                      type: string
-                    version:
-                      type: string
-              provider:
-                type: object
-                properties:
-                  name:
-                    type: string
-                  url:
-                    type: string
-              replaces:
-                description: The name of a CSV this one replaces. Should match the
-                  `metadata.Name` field of the old CSV.
-                type: string
-              selector:
-                description: Label selector for related resources.
-                type: object
-                properties:
-                  matchExpressions:
-                    description: matchExpressions is a list of label selector requirements.
-                      The requirements are ANDed.
-                    type: array
-                    items:
-                      description: A label selector requirement is a selector that
-                        contains values, a key, and an operator that relates the key
-                        and values.
-                      type: object
-                      required:
-                      - key
-                      - operator
-                      properties:
-                        key:
-                          description: key is the label key that the selector applies
-                            to.
-                          type: string
-                        operator:
-                          description: operator represents a key's relationship to
-                            a set of values. Valid operators are In, NotIn, Exists
-                            and DoesNotExist.
-                          type: string
-                        values:
-                          description: values is an array of string values. If the
-                            operator is In or NotIn, the values array must be non-empty.
-                            If the operator is Exists or DoesNotExist, the values
-                            array must be empty. This array is replaced during a strategic
-                            merge patch.
-                          type: array
-                          items:
-                            type: string
-                  matchLabels:
-                    description: matchLabels is a map of {key,value} pairs. A single
-                      {key,value} in the matchLabels map is equivalent to an element
-                      of matchExpressions, whose key field is "key", the operator
-                      is "In", and the values array contains only "value". The requirements
-                      are ANDed.
-                    type: object
-                    additionalProperties:
-                      type: string
-              version:
-                description: OperatorVersion is a wrapper around semver.Version which
-                  supports correct marshaling to YAML and JSON.
-                type: string
-              webhookdefinitions:
-                type: array
-                items:
-                  description: WebhookDescription provides details to OLM about required
-                    webhooks
-                  type: object
-                  required:
-                  - admissionReviewVersions
-                  - generateName
-                  - sideEffects
-                  - type
-                  properties:
-                    admissionReviewVersions:
-                      type: array
-                      items:
-                        type: string
-                    containerPort:
-                      type: integer
-                      format: int32
-                      default: 443
-                      maximum: 65535
-                      minimum: 1
-                    conversionCRDs:
-                      type: array
-                      items:
-                        type: string
-                    deploymentName:
-                      type: string
-                    failurePolicy:
-                      type: string
-                    generateName:
-                      type: string
-                    matchPolicy:
-                      description: MatchPolicyType specifies the type of match policy
-                      type: string
-                    objectSelector:
-                      description: A label selector is a label query over a set of
-                        resources. The result of matchLabels and matchExpressions
-                        are ANDed. An empty label selector matches all objects. A
-                        null label selector matches no objects.
-                      type: object
-                      properties:
-                        matchExpressions:
-                          description: matchExpressions is a list of label selector
-                            requirements. The requirements are ANDed.
-                          type: array
-                          items:
-                            description: A label selector requirement is a selector
-                              that contains values, a key, and an operator that relates
-                              the key and values.
-                            type: object
-                            required:
-                            - key
-                            - operator
-                            properties:
-                              key:
-                                description: key is the label key that the selector
-                                  applies to.
-                                type: string
-                              operator:
-                                description: operator represents a key's relationship
-                                  to a set of values. Valid operators are In, NotIn,
-                                  Exists and DoesNotExist.
-                                type: string
-                              values:
-                                description: values is an array of string values.
-                                  If the operator is In or NotIn, the values array
-                                  must be non-empty. If the operator is Exists or
-                                  DoesNotExist, the values array must be empty. This
-                                  array is replaced during a strategic merge patch.
-                                type: array
-                                items:
-                                  type: string
-                        matchLabels:
-                          description: matchLabels is a map of {key,value} pairs.
-                            A single {key,value} in the matchLabels map is equivalent
-                            to an element of matchExpressions, whose key field is
-                            "key", the operator is "In", and the values array contains
-                            only "value". The requirements are ANDed.
-                          type: object
-                          additionalProperties:
-                            type: string
-                    reinvocationPolicy:
-                      description: ReinvocationPolicyType specifies what type of policy
-                        the admission hook uses.
-                      type: string
-                    rules:
-                      type: array
-                      items:
-                        description: RuleWithOperations is a tuple of Operations and
-                          Resources. It is recommended to make sure that all the tuple
-                          expansions are valid.
-                        type: object
-                        properties:
-                          apiGroups:
-                            description: APIGroups is the API groups the resources
-                              belong to. '*' is all groups. If '*' is present, the
-                              length of the slice must be one. Required.
-                            type: array
-                            items:
-                              type: string
-                          apiVersions:
-                            description: APIVersions is the API versions the resources
-                              belong to. '*' is all versions. If '*' is present, the
-                              length of the slice must be one. Required.
-                            type: array
-                            items:
-                              type: string
-                          operations:
-                            description: Operations is the operations the admission
-                              hook cares about - CREATE, UPDATE, or * for all operations.
-                              If '*' is present, the length of the slice must be one.
-                              Required.
-                            type: array
-                            items:
-                              type: string
-                          resources:
-                            description: "Resources is a list of resources this rule
-                              applies to. \n For example: 'pods' means pods. 'pods/log'
-                              means the log subresource of pods. '*' means all resources,
-                              but not subresources. 'pods/*' means all subresources
-                              of pods. '*/scale' means all scale subresources. '*/*'
-                              means all resources and their subresources. \n If wildcard
-                              is present, the validation rule will ensure resources
-                              do not overlap with each other. \n Depending on the
-                              enclosing object, subresources might not be allowed.
-                              Required."
-                            type: array
-                            items:
-                              type: string
-                          scope:
-                            description: scope specifies the scope of this rule. Valid
-                              values are "Cluster", "Namespaced", and "*" "Cluster"
-                              means that only cluster-scoped resources will match
-                              this rule. Namespace API objects are cluster-scoped.
-                              "Namespaced" means that only namespaced resources will
-                              match this rule. "*" means that there are no scope restrictions.
-                              Subresources match the scope of their parent resource.
-                              Default is "*".
-                            type: string
-                    sideEffects:
-                      type: string
-                    targetPort:
-                      anyOf:
-                      - type: integer
-                      - type: string
-                      x-kubernetes-int-or-string: true
-                    timeoutSeconds:
-                      type: integer
-                      format: int32
-                    type:
-                      description: WebhookAdmissionType is the type of admission webhooks
-                        supported by OLM
-                      type: string
-                      enum:
-                      - ValidatingAdmissionWebhook
-                      - MutatingAdmissionWebhook
-                      - ConversionWebhook
-                    webhookPath:
-                      type: string
-          status:
-            description: ClusterServiceVersionStatus represents information about
-              the status of a pod. Status may trail the actual state of a system.
-            type: object
-            properties:
-              certsLastUpdated:
-                description: Last time the owned APIService certs were updated
-                type: string
-                format: date-time
-              certsRotateAt:
-                description: Time the owned APIService certs will rotate next
-                type: string
-                format: date-time
-              conditions:
-                description: List of conditions, a history of state transitions
-                type: array
-                items:
-                  description: Conditions appear in the status as a record of state
-                    transitions on the ClusterServiceVersion
-                  type: object
-                  properties:
-                    lastTransitionTime:
-                      description: Last time the status transitioned from one status
-                        to another.
-                      type: string
-                      format: date-time
-                    lastUpdateTime:
-                      description: Last time we updated the status
-                      type: string
-                      format: date-time
-                    message:
-                      description: A human readable message indicating details about
-                        why the ClusterServiceVersion is in this condition.
-                      type: string
-                    phase:
-                      description: Condition of the ClusterServiceVersion
-                      type: string
-                    reason:
-                      description: A brief CamelCase message indicating details about
-                        why the ClusterServiceVersion is in this state. e.g. 'RequirementsNotMet'
-                      type: string
-              lastTransitionTime:
-                description: Last time the status transitioned from one status to
-                  another.
-                type: string
-                format: date-time
-              lastUpdateTime:
-                description: Last time we updated the status
-                type: string
-                format: date-time
-              message:
-                description: A human readable message indicating details about why
-                  the ClusterServiceVersion is in this condition.
-                type: string
-              phase:
-                description: Current condition of the ClusterServiceVersion
-                type: string
-              reason:
-                description: A brief CamelCase message indicating details about why
-                  the ClusterServiceVersion is in this state. e.g. 'RequirementsNotMet'
-                type: string
-              requirementStatus:
-                description: The status of each requirement for this CSV
-                type: array
-                items:
-                  type: object
-                  required:
-                  - group
-                  - kind
-                  - message
-                  - name
-                  - status
-                  - version
-                  properties:
-                    dependents:
-                      type: array
-                      items:
-                        description: DependentStatus is the status for a dependent
-                          requirement (to prevent infinite nesting)
-                        type: object
-                        required:
-                        - group
-                        - kind
-                        - status
-                        - version
-                        properties:
-                          group:
-                            type: string
-                          kind:
-                            type: string
-                          message:
-                            type: string
-                          status:
-                            description: StatusReason is a camelcased reason for the
-                              status of a RequirementStatus or DependentStatus
-                            type: string
-                          uuid:
-                            type: string
-                          version:
-                            type: string
-                    group:
-                      type: string
-                    kind:
-                      type: string
-                    message:
-                      type: string
-                    name:
-                      type: string
-                    status:
-                      description: StatusReason is a camelcased reason for the status
-                        of a RequirementStatus or DependentStatus
-                      type: string
-                    uuid:
-                      type: string
-                    version:
-                      type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: installplans.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: InstallPlan
-    listKind: InstallPlanList
-    plural: installplans
-    shortNames:
-    - ip
-    singular: installplan
-  scope: Namespaced
-  versions:
-  - additionalPrinterColumns:
-    - description: The first CSV in the list of clusterServiceVersionNames
-      jsonPath: .spec.clusterServiceVersionNames[0]
-      name: CSV
-      type: string
-    - description: The approval mode
-      jsonPath: .spec.approval
-      name: Approval
-      type: string
-    - jsonPath: .spec.approved
-      name: Approved
-      type: boolean
-    name: v1alpha1
-    schema:
-      openAPIV3Schema:
-        description: InstallPlan defines the installation of a set of operators.
-        type: object
-        required:
-        - metadata
-        - spec
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: InstallPlanSpec defines a set of Application resources to
-              be installed
-            type: object
-            required:
-            - approval
-            - approved
-            - clusterServiceVersionNames
-            properties:
-              approval:
-                description: Approval is the user approval policy for an InstallPlan.
-                  It must be one of "Automatic" or "Manual".
-                type: string
-              approved:
-                type: boolean
-              clusterServiceVersionNames:
-                type: array
-                items:
-                  type: string
-              generation:
-                type: integer
-              source:
-                type: string
-              sourceNamespace:
-                type: string
-          status:
-            description: "InstallPlanStatus represents the information about the status
-              of steps required to complete installation. \n Status may trail the
-              actual state of a system."
-            type: object
-            required:
-            - catalogSources
-            - phase
-            properties:
-              attenuatedServiceAccountRef:
-                description: AttenuatedServiceAccountRef references the service account
-                  that is used to do scoped operator install.
-                type: object
-                properties:
-                  apiVersion:
-                    description: API version of the referent.
-                    type: string
-                  fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
-                    type: string
-                  kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                    type: string
-                  name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                    type: string
-                  namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                    type: string
-                  resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                    type: string
-                  uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                    type: string
-              bundleLookups:
-                description: BundleLookups is the set of in-progress requests to pull
-                  and unpackage bundle content to the cluster.
-                type: array
-                items:
-                  description: BundleLookup is a request to pull and unpackage the
-                    content of a bundle to the cluster.
-                  type: object
-                  required:
-                  - catalogSourceRef
-                  - identifier
-                  - path
-                  - replaces
-                  properties:
-                    catalogSourceRef:
-                      description: CatalogSourceRef is a reference to the CatalogSource
-                        the bundle path was resolved from.
-                      type: object
-                      properties:
-                        apiVersion:
-                          description: API version of the referent.
-                          type: string
-                        fieldPath:
-                          description: 'If referring to a piece of an object instead
-                            of an entire object, this string should contain a valid
-                            JSON/Go field access statement, such as desiredState.manifest.containers[2].
-                            For example, if the object reference is to a container
-                            within a pod, this would take on a value like: "spec.containers{name}"
-                            (where "name" refers to the name of the container that
-                            triggered the event) or if no container name is specified
-                            "spec.containers[2]" (container with index 2 in this pod).
-                            This syntax is chosen only to have some well-defined way
-                            of referencing a part of an object. TODO: this design
-                            is not final and this field is subject to change in the
-                            future.'
-                          type: string
-                        kind:
-                          description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                          type: string
-                        name:
-                          description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                          type: string
-                        namespace:
-                          description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                          type: string
-                        resourceVersion:
-                          description: 'Specific resourceVersion to which this reference
-                            is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                          type: string
-                        uid:
-                          description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                          type: string
-                    conditions:
-                      description: Conditions represents the overall state of a BundleLookup.
-                      type: array
-                      items:
-                        type: object
-                        required:
-                        - status
-                        - type
-                        properties:
-                          lastTransitionTime:
-                            description: Last time the condition transitioned from
-                              one status to another.
-                            type: string
-                            format: date-time
-                          lastUpdateTime:
-                            description: Last time the condition was probed.
-                            type: string
-                            format: date-time
-                          message:
-                            description: A human readable message indicating details
-                              about the transition.
-                            type: string
-                          reason:
-                            description: The reason for the condition's last transition.
-                            type: string
-                          status:
-                            description: Status of the condition, one of True, False,
-                              Unknown.
-                            type: string
-                          type:
-                            description: Type of condition.
-                            type: string
-                    identifier:
-                      description: Identifier is the catalog-unique name of the operator
-                        (the name of the CSV for bundles that contain CSVs)
-                      type: string
-                    path:
-                      description: Path refers to the location of a bundle to pull.
-                        It's typically an image reference.
-                      type: string
-                    properties:
-                      description: The effective properties of the unpacked bundle.
-                      type: string
-                    replaces:
-                      description: Replaces is the name of the bundle to replace with
-                        the one found at Path.
-                      type: string
-              catalogSources:
-                type: array
-                items:
-                  type: string
-              conditions:
-                type: array
-                items:
-                  description: InstallPlanCondition represents the overall status
-                    of the execution of an InstallPlan.
-                  type: object
-                  properties:
-                    lastTransitionTime:
-                      type: string
-                      format: date-time
-                    lastUpdateTime:
-                      type: string
-                      format: date-time
-                    message:
-                      type: string
-                    reason:
-                      description: ConditionReason is a camelcased reason for the
-                        state transition.
-                      type: string
-                    status:
-                      type: string
-                    type:
-                      description: InstallPlanConditionType describes the state of
-                        an InstallPlan at a certain point as a whole.
-                      type: string
-              phase:
-                description: InstallPlanPhase is the current status of a InstallPlan
-                  as a whole.
-                type: string
-              plan:
-                type: array
-                items:
-                  description: Step represents the status of an individual step in
-                    an InstallPlan.
-                  type: object
-                  required:
-                  - resolving
-                  - resource
-                  - status
-                  properties:
-                    resolving:
-                      type: string
-                    resource:
-                      description: StepResource represents the status of a resource
-                        to be tracked by an InstallPlan.
-                      type: object
-                      required:
-                      - group
-                      - kind
-                      - name
-                      - sourceName
-                      - sourceNamespace
-                      - version
-                      properties:
-                        group:
-                          type: string
-                        kind:
-                          type: string
-                        manifest:
-                          type: string
-                        name:
-                          type: string
-                        sourceName:
-                          type: string
-                        sourceNamespace:
-                          type: string
-                        version:
-                          type: string
-                    status:
-                      description: StepStatus is the current status of a particular
-                        resource an in InstallPlan
-                      type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: operatorgroups.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: OperatorGroup
-    listKind: OperatorGroupList
-    plural: operatorgroups
-    shortNames:
-    - og
-    singular: operatorgroup
-  scope: Namespaced
-  versions:
-  - name: v1
-    schema:
-      openAPIV3Schema:
-        description: OperatorGroup is the unit of multitenancy for OLM managed operators.
-          It constrains the installation of operators in its namespace to a specified
-          set of target namespaces.
-        type: object
-        required:
-        - metadata
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: OperatorGroupSpec is the spec for an OperatorGroup resource.
-            type: object
-            properties:
-              selector:
-                description: Selector selects the OperatorGroup's target namespaces.
-                type: object
-                properties:
-                  matchExpressions:
-                    description: matchExpressions is a list of label selector requirements.
-                      The requirements are ANDed.
-                    type: array
-                    items:
-                      description: A label selector requirement is a selector that
-                        contains values, a key, and an operator that relates the key
-                        and values.
-                      type: object
-                      required:
-                      - key
-                      - operator
-                      properties:
-                        key:
-                          description: key is the label key that the selector applies
-                            to.
-                          type: string
-                        operator:
-                          description: operator represents a key's relationship to
-                            a set of values. Valid operators are In, NotIn, Exists
-                            and DoesNotExist.
-                          type: string
-                        values:
-                          description: values is an array of string values. If the
-                            operator is In or NotIn, the values array must be non-empty.
-                            If the operator is Exists or DoesNotExist, the values
-                            array must be empty. This array is replaced during a strategic
-                            merge patch.
-                          type: array
-                          items:
-                            type: string
-                  matchLabels:
-                    description: matchLabels is a map of {key,value} pairs. A single
-                      {key,value} in the matchLabels map is equivalent to an element
-                      of matchExpressions, whose key field is "key", the operator
-                      is "In", and the values array contains only "value". The requirements
-                      are ANDed.
-                    type: object
-                    additionalProperties:
-                      type: string
-              serviceAccountName:
-                description: ServiceAccountName is the admin specified service account
-                  which will be used to deploy operator(s) in this operator group.
-                type: string
-              staticProvidedAPIs:
-                description: Static tells OLM not to update the OperatorGroup's providedAPIs
-                  annotation
-                type: boolean
-              targetNamespaces:
-                description: TargetNamespaces is an explicit set of namespaces to
-                  target. If it is set, Selector is ignored.
-                type: array
-                items:
-                  type: string
-                x-kubernetes-list-type: set
-          status:
-            description: OperatorGroupStatus is the status for an OperatorGroupResource.
-            type: object
-            required:
-            - lastUpdated
-            properties:
-              lastUpdated:
-                description: LastUpdated is a timestamp of the last time the OperatorGroup's
-                  status was Updated.
-                type: string
-                format: date-time
-              namespaces:
-                description: Namespaces is the set of target namespaces for the OperatorGroup.
-                type: array
-                items:
-                  type: string
-                x-kubernetes-list-type: set
-              serviceAccountRef:
-                description: ServiceAccountRef references the service account object
-                  specified.
-                type: object
-                properties:
-                  apiVersion:
-                    description: API version of the referent.
-                    type: string
-                  fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
-                    type: string
-                  kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                    type: string
-                  name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                    type: string
-                  namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                    type: string
-                  resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                    type: string
-                  uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                    type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-  - name: v1alpha2
-    schema:
-      openAPIV3Schema:
-        description: OperatorGroup is the unit of multitenancy for OLM managed operators.
-          It constrains the installation of operators in its namespace to a specified
-          set of target namespaces.
-        type: object
-        required:
-        - metadata
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: OperatorGroupSpec is the spec for an OperatorGroup resource.
-            type: object
-            properties:
-              selector:
-                description: Selector selects the OperatorGroup's target namespaces.
-                type: object
-                properties:
-                  matchExpressions:
-                    description: matchExpressions is a list of label selector requirements.
-                      The requirements are ANDed.
-                    type: array
-                    items:
-                      description: A label selector requirement is a selector that
-                        contains values, a key, and an operator that relates the key
-                        and values.
-                      type: object
-                      required:
-                      - key
-                      - operator
-                      properties:
-                        key:
-                          description: key is the label key that the selector applies
-                            to.
-                          type: string
-                        operator:
-                          description: operator represents a key's relationship to
-                            a set of values. Valid operators are In, NotIn, Exists
-                            and DoesNotExist.
-                          type: string
-                        values:
-                          description: values is an array of string values. If the
-                            operator is In or NotIn, the values array must be non-empty.
-                            If the operator is Exists or DoesNotExist, the values
-                            array must be empty. This array is replaced during a strategic
-                            merge patch.
-                          type: array
-                          items:
-                            type: string
-                  matchLabels:
-                    description: matchLabels is a map of {key,value} pairs. A single
-                      {key,value} in the matchLabels map is equivalent to an element
-                      of matchExpressions, whose key field is "key", the operator
-                      is "In", and the values array contains only "value". The requirements
-                      are ANDed.
-                    type: object
-                    additionalProperties:
-                      type: string
-              serviceAccountName:
-                description: ServiceAccountName is the admin specified service account
-                  which will be used to deploy operator(s) in this operator group.
-                type: string
-              staticProvidedAPIs:
-                description: Static tells OLM not to update the OperatorGroup's providedAPIs
-                  annotation
-                type: boolean
-              targetNamespaces:
-                description: TargetNamespaces is an explicit set of namespaces to
-                  target. If it is set, Selector is ignored.
-                type: array
-                items:
-                  type: string
-          status:
-            description: OperatorGroupStatus is the status for an OperatorGroupResource.
-            type: object
-            required:
-            - lastUpdated
-            properties:
-              lastUpdated:
-                description: LastUpdated is a timestamp of the last time the OperatorGroup's
-                  status was Updated.
-                type: string
-                format: date-time
-              namespaces:
-                description: Namespaces is the set of target namespaces for the OperatorGroup.
-                type: array
-                items:
-                  type: string
-              serviceAccountRef:
-                description: ServiceAccountRef references the service account object
-                  specified.
-                type: object
-                properties:
-                  apiVersion:
-                    description: API version of the referent.
-                    type: string
-                  fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
-                    type: string
-                  kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                    type: string
-                  name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                    type: string
-                  namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                    type: string
-                  resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                    type: string
-                  uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                    type: string
-    served: true
-    storage: false
-    subresources:
-      status: {}
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: operators.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: Operator
-    listKind: OperatorList
-    plural: operators
-    singular: operator
-  scope: Cluster
-  versions:
-  - name: v1
-    schema:
-      openAPIV3Schema:
-        description: Operator represents a cluster operator.
-        type: object
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: OperatorSpec defines the desired state of Operator
-            type: object
-          status:
-            description: OperatorStatus defines the observed state of an Operator
-              and its components
-            type: object
-            properties:
-              components:
-                description: Components describes resources that compose the operator.
-                type: object
-                required:
-                - labelSelector
-                properties:
-                  labelSelector:
-                    description: LabelSelector is a label query over a set of resources
-                      used to select the operator's components
-                    type: object
-                    properties:
-                      matchExpressions:
-                        description: matchExpressions is a list of label selector
-                          requirements. The requirements are ANDed.
-                        type: array
-                        items:
-                          description: A label selector requirement is a selector
-                            that contains values, a key, and an operator that relates
-                            the key and values.
-                          type: object
-                          required:
-                          - key
-                          - operator
-                          properties:
-                            key:
-                              description: key is the label key that the selector
-                                applies to.
-                              type: string
-                            operator:
-                              description: operator represents a key's relationship
-                                to a set of values. Valid operators are In, NotIn,
-                                Exists and DoesNotExist.
-                              type: string
-                            values:
-                              description: values is an array of string values. If
-                                the operator is In or NotIn, the values array must
-                                be non-empty. If the operator is Exists or DoesNotExist,
-                                the values array must be empty. This array is replaced
-                                during a strategic merge patch.
-                              type: array
-                              items:
-                                type: string
-                      matchLabels:
-                        description: matchLabels is a map of {key,value} pairs. A
-                          single {key,value} in the matchLabels map is equivalent
-                          to an element of matchExpressions, whose key field is "key",
-                          the operator is "In", and the values array contains only
-                          "value". The requirements are ANDed.
-                        type: object
-                        additionalProperties:
-                          type: string
-                  refs:
-                    description: Refs are a set of references to the operator's component
-                      resources, selected with LabelSelector.
-                    type: array
-                    items:
-                      description: RichReference is a reference to a resource, enriched
-                        with its status conditions.
-                      type: object
-                      properties:
-                        apiVersion:
-                          description: API version of the referent.
-                          type: string
-                        conditions:
-                          description: Conditions represents the latest state of the
-                            component.
-                          type: array
-                          items:
-                            description: Condition represent the latest available
-                              observations of an component's state.
-                            type: object
-                            required:
-                            - status
-                            - type
-                            properties:
-                              lastTransitionTime:
-                                description: Last time the condition transitioned
-                                  from one status to another.
-                                type: string
-                                format: date-time
-                              lastUpdateTime:
-                                description: Last time the condition was probed
-                                type: string
-                                format: date-time
-                              message:
-                                description: A human readable message indicating details
-                                  about the transition.
-                                type: string
-                              reason:
-                                description: The reason for the condition's last transition.
-                                type: string
-                              status:
-                                description: Status of the condition, one of True,
-                                  False, Unknown.
-                                type: string
-                              type:
-                                description: Type of condition.
-                                type: string
-                        fieldPath:
-                          description: 'If referring to a piece of an object instead
-                            of an entire object, this string should contain a valid
-                            JSON/Go field access statement, such as desiredState.manifest.containers[2].
-                            For example, if the object reference is to a container
-                            within a pod, this would take on a value like: "spec.containers{name}"
-                            (where "name" refers to the name of the container that
-                            triggered the event) or if no container name is specified
-                            "spec.containers[2]" (container with index 2 in this pod).
-                            This syntax is chosen only to have some well-defined way
-                            of referencing a part of an object. TODO: this design
-                            is not final and this field is subject to change in the
-                            future.'
-                          type: string
-                        kind:
-                          description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                          type: string
-                        name:
-                          description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                          type: string
-                        namespace:
-                          description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                          type: string
-                        resourceVersion:
-                          description: 'Specific resourceVersion to which this reference
-                            is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                          type: string
-                        uid:
-                          description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                          type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  annotations:
-    controller-gen.kubebuilder.io/version: v0.3.0
-  creationTimestamp: null
-  name: subscriptions.operators.coreos.com
-spec:
-  group: operators.coreos.com
-  names:
-    categories:
-    - olm
-    kind: Subscription
-    listKind: SubscriptionList
-    plural: subscriptions
-    shortNames:
-    - sub
-    - subs
-    singular: subscription
-  scope: Namespaced
-  versions:
-  - additionalPrinterColumns:
-    - description: The package subscribed to
-      jsonPath: .spec.name
-      name: Package
-      type: string
-    - description: The catalog source for the specified package
-      jsonPath: .spec.source
-      name: Source
-      type: string
-    - description: The channel of updates to subscribe to
-      jsonPath: .spec.channel
-      name: Channel
-      type: string
-    name: v1alpha1
-    schema:
-      openAPIV3Schema:
-        description: Subscription keeps operators up to date by tracking changes to
-          Catalogs.
-        type: object
-        required:
-        - metadata
-        - spec
-        properties:
-          apiVersion:
-            description: 'APIVersion defines the versioned schema of this representation
-              of an object. Servers should convert recognized schemas to the latest
-              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-            type: string
-          kind:
-            description: 'Kind is a string value representing the REST resource this
-              object represents. Servers may infer this from the endpoint the client
-              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-            type: string
-          metadata:
-            type: object
-          spec:
-            description: SubscriptionSpec defines an Application that can be installed
-            type: object
-            required:
-            - name
-            - source
-            - sourceNamespace
-            properties:
-              channel:
-                type: string
-              config:
-                description: SubscriptionConfig contains configuration specified for
-                  a subscription.
-                type: object
-                properties:
-                  env:
-                    description: Env is a list of environment variables to set in
-                      the container. Cannot be updated.
-                    type: array
-                    items:
-                      description: EnvVar represents an environment variable present
-                        in a Container.
-                      type: object
-                      required:
-                      - name
-                      properties:
-                        name:
-                          description: Name of the environment variable. Must be a
-                            C_IDENTIFIER.
-                          type: string
-                        value:
-                          description: 'Variable references $(VAR_NAME) are expanded
-                            using the previous defined environment variables in the
-                            container and any service environment variables. If a
-                            variable cannot be resolved, the reference in the input
-                            string will be unchanged. The $(VAR_NAME) syntax can be
-                            escaped with a double $$, ie: $$(VAR_NAME). Escaped references
-                            will never be expanded, regardless of whether the variable
-                            exists or not. Defaults to "".'
-                          type: string
-                        valueFrom:
-                          description: Source for the environment variable's value.
-                            Cannot be used if value is not empty.
-                          type: object
-                          properties:
-                            configMapKeyRef:
-                              description: Selects a key of a ConfigMap.
-                              type: object
-                              required:
-                              - key
-                              properties:
-                                key:
-                                  description: The key to select.
-                                  type: string
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                                optional:
-                                  description: Specify whether the ConfigMap or its
-                                    key must be defined
-                                  type: boolean
-                            fieldRef:
-                              description: 'Selects a field of the pod: supports metadata.name,
-                                metadata.namespace, metadata.labels, metadata.annotations,
-                                spec.nodeName, spec.serviceAccountName, status.hostIP,
-                                status.podIP, status.podIPs.'
-                              type: object
-                              required:
-                              - fieldPath
-                              properties:
-                                apiVersion:
-                                  description: Version of the schema the FieldPath
-                                    is written in terms of, defaults to "v1".
-                                  type: string
-                                fieldPath:
-                                  description: Path of the field to select in the
-                                    specified API version.
-                                  type: string
-                            resourceFieldRef:
-                              description: 'Selects a resource of the container: only
-                                resources limits and requests (limits.cpu, limits.memory,
-                                limits.ephemeral-storage, requests.cpu, requests.memory
-                                and requests.ephemeral-storage) are currently supported.'
-                              type: object
-                              required:
-                              - resource
-                              properties:
-                                containerName:
-                                  description: 'Container name: required for volumes,
-                                    optional for env vars'
-                                  type: string
-                                divisor:
-                                  description: Specifies the output format of the
-                                    exposed resources, defaults to "1"
-                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                  anyOf:
-                                  - type: integer
-                                  - type: string
-                                  x-kubernetes-int-or-string: true
-                                resource:
-                                  description: 'Required: resource to select'
-                                  type: string
-                            secretKeyRef:
-                              description: Selects a key of a secret in the pod's
-                                namespace
-                              type: object
-                              required:
-                              - key
-                              properties:
-                                key:
-                                  description: The key of the secret to select from.  Must
-                                    be a valid secret key.
-                                  type: string
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                                optional:
-                                  description: Specify whether the Secret or its key
-                                    must be defined
-                                  type: boolean
-                  envFrom:
-                    description: EnvFrom is a list of sources to populate environment
-                      variables in the container. The keys defined within a source
-                      must be a C_IDENTIFIER. All invalid keys will be reported as
-                      an event when the container is starting. When a key exists in
-                      multiple sources, the value associated with the last source
-                      will take precedence. Values defined by an Env with a duplicate
-                      key will take precedence. Immutable.
-                    type: array
-                    items:
-                      description: EnvFromSource represents the source of a set of
-                        ConfigMaps
-                      type: object
-                      properties:
-                        configMapRef:
-                          description: The ConfigMap to select from
-                          type: object
-                          properties:
-                            name:
-                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                TODO: Add other useful fields. apiVersion, kind, uid?'
-                              type: string
-                            optional:
-                              description: Specify whether the ConfigMap must be defined
-                              type: boolean
-                        prefix:
-                          description: An optional identifier to prepend to each key
-                            in the ConfigMap. Must be a C_IDENTIFIER.
-                          type: string
-                        secretRef:
-                          description: The Secret to select from
-                          type: object
-                          properties:
-                            name:
-                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                TODO: Add other useful fields. apiVersion, kind, uid?'
-                              type: string
-                            optional:
-                              description: Specify whether the Secret must be defined
-                              type: boolean
-                  nodeSelector:
-                    description: 'NodeSelector is a selector which must be true for
-                      the pod to fit on a node. Selector which must match a node''s
-                      labels for the pod to be scheduled on that node. More info:
-                      https://kubernetes.io/docs/concepts/configuration/assign-pod-node/'
-                    type: object
-                    additionalProperties:
-                      type: string
-                  resources:
-                    description: 'Resources represents compute resources required
-                      by this container. Immutable. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                    type: object
-                    properties:
-                      limits:
-                        description: 'Limits describes the maximum amount of compute
-                          resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                        type: object
-                        additionalProperties:
-                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                          anyOf:
-                          - type: integer
-                          - type: string
-                          x-kubernetes-int-or-string: true
-                      requests:
-                        description: 'Requests describes the minimum amount of compute
-                          resources required. If Requests is omitted for a container,
-                          it defaults to Limits if that is explicitly specified, otherwise
-                          to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
-                        type: object
-                        additionalProperties:
-                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                          anyOf:
-                          - type: integer
-                          - type: string
-                          x-kubernetes-int-or-string: true
-                  selector:
-                    description: Selector is the label selector for pods to be configured.
-                      Existing ReplicaSets whose pods are selected by this will be
-                      the ones affected by this deployment. It must match the pod
-                      template's labels.
-                    type: object
-                    properties:
-                      matchExpressions:
-                        description: matchExpressions is a list of label selector
-                          requirements. The requirements are ANDed.
-                        type: array
-                        items:
-                          description: A label selector requirement is a selector
-                            that contains values, a key, and an operator that relates
-                            the key and values.
-                          type: object
-                          required:
-                          - key
-                          - operator
-                          properties:
-                            key:
-                              description: key is the label key that the selector
-                                applies to.
-                              type: string
-                            operator:
-                              description: operator represents a key's relationship
-                                to a set of values. Valid operators are In, NotIn,
-                                Exists and DoesNotExist.
-                              type: string
-                            values:
-                              description: values is an array of string values. If
-                                the operator is In or NotIn, the values array must
-                                be non-empty. If the operator is Exists or DoesNotExist,
-                                the values array must be empty. This array is replaced
-                                during a strategic merge patch.
-                              type: array
-                              items:
-                                type: string
-                      matchLabels:
-                        description: matchLabels is a map of {key,value} pairs. A
-                          single {key,value} in the matchLabels map is equivalent
-                          to an element of matchExpressions, whose key field is "key",
-                          the operator is "In", and the values array contains only
-                          "value". The requirements are ANDed.
-                        type: object
-                        additionalProperties:
-                          type: string
-                  tolerations:
-                    description: Tolerations are the pod's tolerations.
-                    type: array
-                    items:
-                      description: The pod this Toleration is attached to tolerates
-                        any taint that matches the triple <key,value,effect> using
-                        the matching operator <operator>.
-                      type: object
-                      properties:
-                        effect:
-                          description: Effect indicates the taint effect to match.
-                            Empty means match all taint effects. When specified, allowed
-                            values are NoSchedule, PreferNoSchedule and NoExecute.
-                          type: string
-                        key:
-                          description: Key is the taint key that the toleration applies
-                            to. Empty means match all taint keys. If the key is empty,
-                            operator must be Exists; this combination means to match
-                            all values and all keys.
-                          type: string
-                        operator:
-                          description: Operator represents a key's relationship to
-                            the value. Valid operators are Exists and Equal. Defaults
-                            to Equal. Exists is equivalent to wildcard for value,
-                            so that a pod can tolerate all taints of a particular
-                            category.
-                          type: string
-                        tolerationSeconds:
-                          description: TolerationSeconds represents the period of
-                            time the toleration (which must be of effect NoExecute,
-                            otherwise this field is ignored) tolerates the taint.
-                            By default, it is not set, which means tolerate the taint
-                            forever (do not evict). Zero and negative values will
-                            be treated as 0 (evict immediately) by the system.
-                          type: integer
-                          format: int64
-                        value:
-                          description: Value is the taint value the toleration matches
-                            to. If the operator is Exists, the value should be empty,
-                            otherwise just a regular string.
-                          type: string
-                  volumeMounts:
-                    description: List of VolumeMounts to set in the container.
-                    type: array
-                    items:
-                      description: VolumeMount describes a mounting of a Volume within
-                        a container.
-                      type: object
-                      required:
-                      - mountPath
-                      - name
-                      properties:
-                        mountPath:
-                          description: Path within the container at which the volume
-                            should be mounted.  Must not contain ':'.
-                          type: string
-                        mountPropagation:
-                          description: mountPropagation determines how mounts are
-                            propagated from the host to container and the other way
-                            around. When not set, MountPropagationNone is used. This
-                            field is beta in 1.10.
-                          type: string
-                        name:
-                          description: This must match the Name of a Volume.
-                          type: string
-                        readOnly:
-                          description: Mounted read-only if true, read-write otherwise
-                            (false or unspecified). Defaults to false.
-                          type: boolean
-                        subPath:
-                          description: Path within the volume from which the container's
-                            volume should be mounted. Defaults to "" (volume's root).
-                          type: string
-                        subPathExpr:
-                          description: Expanded path within the volume from which
-                            the container's volume should be mounted. Behaves similarly
-                            to SubPath but environment variable references $(VAR_NAME)
-                            are expanded using the container's environment. Defaults
-                            to "" (volume's root). SubPathExpr and SubPath are mutually
-                            exclusive.
-                          type: string
-                  volumes:
-                    description: List of Volumes to set in the podSpec.
-                    type: array
-                    items:
-                      description: Volume represents a named volume in a pod that
-                        may be accessed by any container in the pod.
-                      type: object
-                      required:
-                      - name
-                      properties:
-                        awsElasticBlockStore:
-                          description: 'AWSElasticBlockStore represents an AWS Disk
-                            resource that is attached to a kubelet''s host machine
-                            and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                          type: object
-                          required:
-                          - volumeID
-                          properties:
-                            fsType:
-                              description: 'Filesystem type of the volume that you
-                                want to mount. Tip: Ensure that the filesystem type
-                                is supported by the host operating system. Examples:
-                                "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4"
-                                if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
-                                TODO: how do we prevent errors in the filesystem from
-                                compromising the machine'
-                              type: string
-                            partition:
-                              description: 'The partition in the volume that you want
-                                to mount. If omitted, the default is to mount by volume
-                                name. Examples: For volume /dev/sda1, you specify
-                                the partition as "1". Similarly, the volume partition
-                                for /dev/sda is "0" (or you can leave the property
-                                empty).'
-                              type: integer
-                              format: int32
-                            readOnly:
-                              description: 'Specify "true" to force and set the ReadOnly
-                                property in VolumeMounts to "true". If omitted, the
-                                default is "false". More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                              type: boolean
-                            volumeID:
-                              description: 'Unique ID of the persistent disk resource
-                                in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore'
-                              type: string
-                        azureDisk:
-                          description: AzureDisk represents an Azure Data Disk mount
-                            on the host and bind mount to the pod.
-                          type: object
-                          required:
-                          - diskName
-                          - diskURI
-                          properties:
-                            cachingMode:
-                              description: 'Host Caching mode: None, Read Only, Read
-                                Write.'
-                              type: string
-                            diskName:
-                              description: The Name of the data disk in the blob storage
-                              type: string
-                            diskURI:
-                              description: The URI the data disk in the blob storage
-                              type: string
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Implicitly inferred to be "ext4" if
-                                unspecified.
-                              type: string
-                            kind:
-                              description: 'Expected values Shared: multiple blob
-                                disks per storage account  Dedicated: single blob
-                                disk per storage account  Managed: azure managed data
-                                disk (only in managed availability set). defaults
-                                to shared'
-                              type: string
-                            readOnly:
-                              description: Defaults to false (read/write). ReadOnly
-                                here will force the ReadOnly setting in VolumeMounts.
-                              type: boolean
-                        azureFile:
-                          description: AzureFile represents an Azure File Service
-                            mount on the host and bind mount to the pod.
-                          type: object
-                          required:
-                          - secretName
-                          - shareName
-                          properties:
-                            readOnly:
-                              description: Defaults to false (read/write). ReadOnly
-                                here will force the ReadOnly setting in VolumeMounts.
-                              type: boolean
-                            secretName:
-                              description: the name of secret that contains Azure
-                                Storage Account Name and Key
-                              type: string
-                            shareName:
-                              description: Share Name
-                              type: string
-                        cephfs:
-                          description: CephFS represents a Ceph FS mount on the host
-                            that shares a pod's lifetime
-                          type: object
-                          required:
-                          - monitors
-                          properties:
-                            monitors:
-                              description: 'Required: Monitors is a collection of
-                                Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                              type: array
-                              items:
-                                type: string
-                            path:
-                              description: 'Optional: Used as the mounted root, rather
-                                than the full Ceph tree, default is /'
-                              type: string
-                            readOnly:
-                              description: 'Optional: Defaults to false (read/write).
-                                ReadOnly here will force the ReadOnly setting in VolumeMounts.
-                                More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                              type: boolean
-                            secretFile:
-                              description: 'Optional: SecretFile is the path to key
-                                ring for User, default is /etc/ceph/user.secret More
-                                info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                              type: string
-                            secretRef:
-                              description: 'Optional: SecretRef is reference to the
-                                authentication secret for User, default is empty.
-                                More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            user:
-                              description: 'Optional: User is the rados user name,
-                                default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it'
-                              type: string
-                        cinder:
-                          description: 'Cinder represents a cinder volume attached
-                            and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                          type: object
-                          required:
-                          - volumeID
-                          properties:
-                            fsType:
-                              description: 'Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Examples:
-                                "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4"
-                                if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                              type: string
-                            readOnly:
-                              description: 'Optional: Defaults to false (read/write).
-                                ReadOnly here will force the ReadOnly setting in VolumeMounts.
-                                More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                              type: boolean
-                            secretRef:
-                              description: 'Optional: points to a secret object containing
-                                parameters used to connect to OpenStack.'
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            volumeID:
-                              description: 'volume id used to identify the volume
-                                in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md'
-                              type: string
-                        configMap:
-                          description: ConfigMap represents a configMap that should
-                            populate this volume
-                          type: object
-                          properties:
-                            defaultMode:
-                              description: 'Optional: mode bits to use on created
-                                files by default. Must be a value between 0 and 0777.
-                                Defaults to 0644. Directories within the path are
-                                not affected by this setting. This might be in conflict
-                                with other options that affect the file mode, like
-                                fsGroup, and the result can be other mode bits set.'
-                              type: integer
-                              format: int32
-                            items:
-                              description: If unspecified, each key-value pair in
-                                the Data field of the referenced ConfigMap will be
-                                projected into the volume as a file whose name is
-                                the key and content is the value. If specified, the
-                                listed keys will be projected into the specified paths,
-                                and unlisted keys will not be present. If a key is
-                                specified which is not present in the ConfigMap, the
-                                volume setup will error unless it is marked optional.
-                                Paths must be relative and may not contain the '..'
-                                path or start with '..'.
-                              type: array
-                              items:
-                                description: Maps a string key to a path within a
-                                  volume.
-                                type: object
-                                required:
-                                - key
-                                - path
-                                properties:
-                                  key:
-                                    description: The key to project.
-                                    type: string
-                                  mode:
-                                    description: 'Optional: mode bits to use on this
-                                      file, must be a value between 0 and 0777. If
-                                      not specified, the volume defaultMode will be
-                                      used. This might be in conflict with other options
-                                      that affect the file mode, like fsGroup, and
-                                      the result can be other mode bits set.'
-                                    type: integer
-                                    format: int32
-                                  path:
-                                    description: The relative path of the file to
-                                      map the key to. May not be an absolute path.
-                                      May not contain the path element '..'. May not
-                                      start with the string '..'.
-                                    type: string
-                            name:
-                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                TODO: Add other useful fields. apiVersion, kind, uid?'
-                              type: string
-                            optional:
-                              description: Specify whether the ConfigMap or its keys
-                                must be defined
-                              type: boolean
-                        csi:
-                          description: CSI (Container Storage Interface) represents
-                            storage that is handled by an external CSI driver (Alpha
-                            feature).
-                          type: object
-                          required:
-                          - driver
-                          properties:
-                            driver:
-                              description: Driver is the name of the CSI driver that
-                                handles this volume. Consult with your admin for the
-                                correct name as registered in the cluster.
-                              type: string
-                            fsType:
-                              description: Filesystem type to mount. Ex. "ext4", "xfs",
-                                "ntfs". If not provided, the empty value is passed
-                                to the associated CSI driver which will determine
-                                the default filesystem to apply.
-                              type: string
-                            nodePublishSecretRef:
-                              description: NodePublishSecretRef is a reference to
-                                the secret object containing sensitive information
-                                to pass to the CSI driver to complete the CSI NodePublishVolume
-                                and NodeUnpublishVolume calls. This field is optional,
-                                and  may be empty if no secret is required. If the
-                                secret object contains more than one secret, all secret
-                                references are passed.
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            readOnly:
-                              description: Specifies a read-only configuration for
-                                the volume. Defaults to false (read/write).
-                              type: boolean
-                            volumeAttributes:
-                              description: VolumeAttributes stores driver-specific
-                                properties that are passed to the CSI driver. Consult
-                                your driver's documentation for supported values.
-                              type: object
-                              additionalProperties:
-                                type: string
-                        downwardAPI:
-                          description: DownwardAPI represents downward API about the
-                            pod that should populate this volume
-                          type: object
-                          properties:
-                            defaultMode:
-                              description: 'Optional: mode bits to use on created
-                                files by default. Must be a value between 0 and 0777.
-                                Defaults to 0644. Directories within the path are
-                                not affected by this setting. This might be in conflict
-                                with other options that affect the file mode, like
-                                fsGroup, and the result can be other mode bits set.'
-                              type: integer
-                              format: int32
-                            items:
-                              description: Items is a list of downward API volume
-                                file
-                              type: array
-                              items:
-                                description: DownwardAPIVolumeFile represents information
-                                  to create the file containing the pod field
-                                type: object
-                                required:
-                                - path
-                                properties:
-                                  fieldRef:
-                                    description: 'Required: Selects a field of the
-                                      pod: only annotations, labels, name and namespace
-                                      are supported.'
-                                    type: object
-                                    required:
-                                    - fieldPath
-                                    properties:
-                                      apiVersion:
-                                        description: Version of the schema the FieldPath
-                                          is written in terms of, defaults to "v1".
-                                        type: string
-                                      fieldPath:
-                                        description: Path of the field to select in
-                                          the specified API version.
-                                        type: string
-                                  mode:
-                                    description: 'Optional: mode bits to use on this
-                                      file, must be a value between 0 and 0777. If
-                                      not specified, the volume defaultMode will be
-                                      used. This might be in conflict with other options
-                                      that affect the file mode, like fsGroup, and
-                                      the result can be other mode bits set.'
-                                    type: integer
-                                    format: int32
-                                  path:
-                                    description: 'Required: Path is  the relative
-                                      path name of the file to be created. Must not
-                                      be absolute or contain the ''..'' path. Must
-                                      be utf-8 encoded. The first item of the relative
-                                      path must not start with ''..'''
-                                    type: string
-                                  resourceFieldRef:
-                                    description: 'Selects a resource of the container:
-                                      only resources limits and requests (limits.cpu,
-                                      limits.memory, requests.cpu and requests.memory)
-                                      are currently supported.'
-                                    type: object
-                                    required:
-                                    - resource
-                                    properties:
-                                      containerName:
-                                        description: 'Container name: required for
-                                          volumes, optional for env vars'
-                                        type: string
-                                      divisor:
-                                        description: Specifies the output format of
-                                          the exposed resources, defaults to "1"
-                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                        anyOf:
-                                        - type: integer
-                                        - type: string
-                                        x-kubernetes-int-or-string: true
-                                      resource:
-                                        description: 'Required: resource to select'
-                                        type: string
-                        emptyDir:
-                          description: 'EmptyDir represents a temporary directory
-                            that shares a pod''s lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir'
-                          type: object
-                          properties:
-                            medium:
-                              description: 'What type of storage medium should back
-                                this directory. The default is "" which means to use
-                                the node''s default medium. Must be an empty string
-                                (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir'
-                              type: string
-                            sizeLimit:
-                              description: 'Total amount of local storage required
-                                for this EmptyDir volume. The size limit is also applicable
-                                for memory medium. The maximum usage on memory medium
-                                EmptyDir would be the minimum value between the SizeLimit
-                                specified here and the sum of memory limits of all
-                                containers in a pod. The default is nil which means
-                                that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir'
-                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                              anyOf:
-                              - type: integer
-                              - type: string
-                              x-kubernetes-int-or-string: true
-                        fc:
-                          description: FC represents a Fibre Channel resource that
-                            is attached to a kubelet's host machine and then exposed
-                            to the pod.
-                          type: object
-                          properties:
-                            fsType:
-                              description: 'Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Implicitly inferred to be "ext4" if
-                                unspecified. TODO: how do we prevent errors in the
-                                filesystem from compromising the machine'
-                              type: string
-                            lun:
-                              description: 'Optional: FC target lun number'
-                              type: integer
-                              format: int32
-                            readOnly:
-                              description: 'Optional: Defaults to false (read/write).
-                                ReadOnly here will force the ReadOnly setting in VolumeMounts.'
-                              type: boolean
-                            targetWWNs:
-                              description: 'Optional: FC target worldwide names (WWNs)'
-                              type: array
-                              items:
-                                type: string
-                            wwids:
-                              description: 'Optional: FC volume world wide identifiers
-                                (wwids) Either wwids or combination of targetWWNs
-                                and lun must be set, but not both simultaneously.'
-                              type: array
-                              items:
-                                type: string
-                        flexVolume:
-                          description: FlexVolume represents a generic volume resource
-                            that is provisioned/attached using an exec based plugin.
-                          type: object
-                          required:
-                          - driver
-                          properties:
-                            driver:
-                              description: Driver is the name of the driver to use
-                                for this volume.
-                              type: string
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". The default filesystem depends on FlexVolume
-                                script.
-                              type: string
-                            options:
-                              description: 'Optional: Extra command options if any.'
-                              type: object
-                              additionalProperties:
-                                type: string
-                            readOnly:
-                              description: 'Optional: Defaults to false (read/write).
-                                ReadOnly here will force the ReadOnly setting in VolumeMounts.'
-                              type: boolean
-                            secretRef:
-                              description: 'Optional: SecretRef is reference to the
-                                secret object containing sensitive information to
-                                pass to the plugin scripts. This may be empty if no
-                                secret object is specified. If the secret object contains
-                                more than one secret, all secrets are passed to the
-                                plugin scripts.'
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                        flocker:
-                          description: Flocker represents a Flocker volume attached
-                            to a kubelet's host machine. This depends on the Flocker
-                            control service being running
-                          type: object
-                          properties:
-                            datasetName:
-                              description: Name of the dataset stored as metadata
-                                -> name on the dataset for Flocker should be considered
-                                as deprecated
-                              type: string
-                            datasetUUID:
-                              description: UUID of the dataset. This is unique identifier
-                                of a Flocker dataset
-                              type: string
-                        gcePersistentDisk:
-                          description: 'GCEPersistentDisk represents a GCE Disk resource
-                            that is attached to a kubelet''s host machine and then
-                            exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                          type: object
-                          required:
-                          - pdName
-                          properties:
-                            fsType:
-                              description: 'Filesystem type of the volume that you
-                                want to mount. Tip: Ensure that the filesystem type
-                                is supported by the host operating system. Examples:
-                                "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4"
-                                if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
-                                TODO: how do we prevent errors in the filesystem from
-                                compromising the machine'
-                              type: string
-                            partition:
-                              description: 'The partition in the volume that you want
-                                to mount. If omitted, the default is to mount by volume
-                                name. Examples: For volume /dev/sda1, you specify
-                                the partition as "1". Similarly, the volume partition
-                                for /dev/sda is "0" (or you can leave the property
-                                empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                              type: integer
-                              format: int32
-                            pdName:
-                              description: 'Unique name of the PD resource in GCE.
-                                Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                              type: string
-                            readOnly:
-                              description: 'ReadOnly here will force the ReadOnly
-                                setting in VolumeMounts. Defaults to false. More info:
-                                https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk'
-                              type: boolean
-                        gitRepo:
-                          description: 'GitRepo represents a git repository at a particular
-                            revision. DEPRECATED: GitRepo is deprecated. To provision
-                            a container with a git repo, mount an EmptyDir into an
-                            InitContainer that clones the repo using git, then mount
-                            the EmptyDir into the Pod''s container.'
-                          type: object
-                          required:
-                          - repository
-                          properties:
-                            directory:
-                              description: Target directory name. Must not contain
-                                or start with '..'.  If '.' is supplied, the volume
-                                directory will be the git repository.  Otherwise,
-                                if specified, the volume will contain the git repository
-                                in the subdirectory with the given name.
-                              type: string
-                            repository:
-                              description: Repository URL
-                              type: string
-                            revision:
-                              description: Commit hash for the specified revision.
-                              type: string
-                        glusterfs:
-                          description: 'Glusterfs represents a Glusterfs mount on
-                            the host that shares a pod''s lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md'
-                          type: object
-                          required:
-                          - endpoints
-                          - path
-                          properties:
-                            endpoints:
-                              description: 'EndpointsName is the endpoint name that
-                                details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                              type: string
-                            path:
-                              description: 'Path is the Glusterfs volume path. More
-                                info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                              type: string
-                            readOnly:
-                              description: 'ReadOnly here will force the Glusterfs
-                                volume to be mounted with read-only permissions. Defaults
-                                to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod'
-                              type: boolean
-                        hostPath:
-                          description: 'HostPath represents a pre-existing file or
-                            directory on the host machine that is directly exposed
-                            to the container. This is generally used for system agents
-                            or other privileged things that are allowed to see the
-                            host machine. Most containers will NOT need this. More
-                            info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
-                            --- TODO(jonesdl) We need to restrict who can use host
-                            directory mounts and who can/can not mount host directories
-                            as read/write.'
-                          type: object
-                          required:
-                          - path
-                          properties:
-                            path:
-                              description: 'Path of the directory on the host. If
-                                the path is a symlink, it will follow the link to
-                                the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath'
-                              type: string
-                            type:
-                              description: 'Type for HostPath Volume Defaults to ""
-                                More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath'
-                              type: string
-                        iscsi:
-                          description: 'ISCSI represents an ISCSI Disk resource that
-                            is attached to a kubelet''s host machine and then exposed
-                            to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md'
-                          type: object
-                          required:
-                          - iqn
-                          - lun
-                          - targetPortal
-                          properties:
-                            chapAuthDiscovery:
-                              description: whether support iSCSI Discovery CHAP authentication
-                              type: boolean
-                            chapAuthSession:
-                              description: whether support iSCSI Session CHAP authentication
-                              type: boolean
-                            fsType:
-                              description: 'Filesystem type of the volume that you
-                                want to mount. Tip: Ensure that the filesystem type
-                                is supported by the host operating system. Examples:
-                                "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4"
-                                if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi
-                                TODO: how do we prevent errors in the filesystem from
-                                compromising the machine'
-                              type: string
-                            initiatorName:
-                              description: Custom iSCSI Initiator Name. If initiatorName
-                                is specified with iscsiInterface simultaneously, new
-                                iSCSI interface <target portal>:<volume name> will
-                                be created for the connection.
-                              type: string
-                            iqn:
-                              description: Target iSCSI Qualified Name.
-                              type: string
-                            iscsiInterface:
-                              description: iSCSI Interface Name that uses an iSCSI
-                                transport. Defaults to 'default' (tcp).
-                              type: string
-                            lun:
-                              description: iSCSI Target Lun number.
-                              type: integer
-                              format: int32
-                            portals:
-                              description: iSCSI Target Portal List. The portal is
-                                either an IP or ip_addr:port if the port is other
-                                than default (typically TCP ports 860 and 3260).
-                              type: array
-                              items:
-                                type: string
-                            readOnly:
-                              description: ReadOnly here will force the ReadOnly setting
-                                in VolumeMounts. Defaults to false.
-                              type: boolean
-                            secretRef:
-                              description: CHAP Secret for iSCSI target and initiator
-                                authentication
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            targetPortal:
-                              description: iSCSI Target Portal. The Portal is either
-                                an IP or ip_addr:port if the port is other than default
-                                (typically TCP ports 860 and 3260).
-                              type: string
-                        name:
-                          description: 'Volume''s name. Must be a DNS_LABEL and unique
-                            within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                          type: string
-                        nfs:
-                          description: 'NFS represents an NFS mount on the host that
-                            shares a pod''s lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                          type: object
-                          required:
-                          - path
-                          - server
-                          properties:
-                            path:
-                              description: 'Path that is exported by the NFS server.
-                                More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                              type: string
-                            readOnly:
-                              description: 'ReadOnly here will force the NFS export
-                                to be mounted with read-only permissions. Defaults
-                                to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                              type: boolean
-                            server:
-                              description: 'Server is the hostname or IP address of
-                                the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs'
-                              type: string
-                        persistentVolumeClaim:
-                          description: 'PersistentVolumeClaimVolumeSource represents
-                            a reference to a PersistentVolumeClaim in the same namespace.
-                            More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
-                          type: object
-                          required:
-                          - claimName
-                          properties:
-                            claimName:
-                              description: 'ClaimName is the name of a PersistentVolumeClaim
-                                in the same namespace as the pod using this volume.
-                                More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims'
-                              type: string
-                            readOnly:
-                              description: Will force the ReadOnly setting in VolumeMounts.
-                                Default false.
-                              type: boolean
-                        photonPersistentDisk:
-                          description: PhotonPersistentDisk represents a PhotonController
-                            persistent disk attached and mounted on kubelets host
-                            machine
-                          type: object
-                          required:
-                          - pdID
-                          properties:
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Implicitly inferred to be "ext4" if
-                                unspecified.
-                              type: string
-                            pdID:
-                              description: ID that identifies Photon Controller persistent
-                                disk
-                              type: string
-                        portworxVolume:
-                          description: PortworxVolume represents a portworx volume
-                            attached and mounted on kubelets host machine
-                          type: object
-                          required:
-                          - volumeID
-                          properties:
-                            fsType:
-                              description: FSType represents the filesystem type to
-                                mount Must be a filesystem type supported by the host
-                                operating system. Ex. "ext4", "xfs". Implicitly inferred
-                                to be "ext4" if unspecified.
-                              type: string
-                            readOnly:
-                              description: Defaults to false (read/write). ReadOnly
-                                here will force the ReadOnly setting in VolumeMounts.
-                              type: boolean
-                            volumeID:
-                              description: VolumeID uniquely identifies a Portworx
-                                volume
-                              type: string
-                        projected:
-                          description: Items for all in one resources secrets, configmaps,
-                            and downward API
-                          type: object
-                          required:
-                          - sources
-                          properties:
-                            defaultMode:
-                              description: Mode bits to use on created files by default.
-                                Must be a value between 0 and 0777. Directories within
-                                the path are not affected by this setting. This might
-                                be in conflict with other options that affect the
-                                file mode, like fsGroup, and the result can be other
-                                mode bits set.
-                              type: integer
-                              format: int32
-                            sources:
-                              description: list of volume projections
-                              type: array
-                              items:
-                                description: Projection that may be projected along
-                                  with other supported volume types
-                                type: object
-                                properties:
-                                  configMap:
-                                    description: information about the configMap data
-                                      to project
-                                    type: object
-                                    properties:
-                                      items:
-                                        description: If unspecified, each key-value
-                                          pair in the Data field of the referenced
-                                          ConfigMap will be projected into the volume
-                                          as a file whose name is the key and content
-                                          is the value. If specified, the listed keys
-                                          will be projected into the specified paths,
-                                          and unlisted keys will not be present. If
-                                          a key is specified which is not present
-                                          in the ConfigMap, the volume setup will
-                                          error unless it is marked optional. Paths
-                                          must be relative and may not contain the
-                                          '..' path or start with '..'.
-                                        type: array
-                                        items:
-                                          description: Maps a string key to a path
-                                            within a volume.
-                                          type: object
-                                          required:
-                                          - key
-                                          - path
-                                          properties:
-                                            key:
-                                              description: The key to project.
-                                              type: string
-                                            mode:
-                                              description: 'Optional: mode bits to
-                                                use on this file, must be a value
-                                                between 0 and 0777. If not specified,
-                                                the volume defaultMode will be used.
-                                                This might be in conflict with other
-                                                options that affect the file mode,
-                                                like fsGroup, and the result can be
-                                                other mode bits set.'
-                                              type: integer
-                                              format: int32
-                                            path:
-                                              description: The relative path of the
-                                                file to map the key to. May not be
-                                                an absolute path. May not contain
-                                                the path element '..'. May not start
-                                                with the string '..'.
-                                              type: string
-                                      name:
-                                        description: 'Name of the referent. More info:
-                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                          TODO: Add other useful fields. apiVersion,
-                                          kind, uid?'
-                                        type: string
-                                      optional:
-                                        description: Specify whether the ConfigMap
-                                          or its keys must be defined
-                                        type: boolean
-                                  downwardAPI:
-                                    description: information about the downwardAPI
-                                      data to project
-                                    type: object
-                                    properties:
-                                      items:
-                                        description: Items is a list of DownwardAPIVolume
-                                          file
-                                        type: array
-                                        items:
-                                          description: DownwardAPIVolumeFile represents
-                                            information to create the file containing
-                                            the pod field
-                                          type: object
-                                          required:
-                                          - path
-                                          properties:
-                                            fieldRef:
-                                              description: 'Required: Selects a field
-                                                of the pod: only annotations, labels,
-                                                name and namespace are supported.'
-                                              type: object
-                                              required:
-                                              - fieldPath
-                                              properties:
-                                                apiVersion:
-                                                  description: Version of the schema
-                                                    the FieldPath is written in terms
-                                                    of, defaults to "v1".
-                                                  type: string
-                                                fieldPath:
-                                                  description: Path of the field to
-                                                    select in the specified API version.
-                                                  type: string
-                                            mode:
-                                              description: 'Optional: mode bits to
-                                                use on this file, must be a value
-                                                between 0 and 0777. If not specified,
-                                                the volume defaultMode will be used.
-                                                This might be in conflict with other
-                                                options that affect the file mode,
-                                                like fsGroup, and the result can be
-                                                other mode bits set.'
-                                              type: integer
-                                              format: int32
-                                            path:
-                                              description: 'Required: Path is  the
-                                                relative path name of the file to
-                                                be created. Must not be absolute or
-                                                contain the ''..'' path. Must be utf-8
-                                                encoded. The first item of the relative
-                                                path must not start with ''..'''
-                                              type: string
-                                            resourceFieldRef:
-                                              description: 'Selects a resource of
-                                                the container: only resources limits
-                                                and requests (limits.cpu, limits.memory,
-                                                requests.cpu and requests.memory)
-                                                are currently supported.'
-                                              type: object
-                                              required:
-                                              - resource
-                                              properties:
-                                                containerName:
-                                                  description: 'Container name: required
-                                                    for volumes, optional for env
-                                                    vars'
-                                                  type: string
-                                                divisor:
-                                                  description: Specifies the output
-                                                    format of the exposed resources,
-                                                    defaults to "1"
-                                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                                                  anyOf:
-                                                  - type: integer
-                                                  - type: string
-                                                  x-kubernetes-int-or-string: true
-                                                resource:
-                                                  description: 'Required: resource
-                                                    to select'
-                                                  type: string
-                                  secret:
-                                    description: information about the secret data
-                                      to project
-                                    type: object
-                                    properties:
-                                      items:
-                                        description: If unspecified, each key-value
-                                          pair in the Data field of the referenced
-                                          Secret will be projected into the volume
-                                          as a file whose name is the key and content
-                                          is the value. If specified, the listed keys
-                                          will be projected into the specified paths,
-                                          and unlisted keys will not be present. If
-                                          a key is specified which is not present
-                                          in the Secret, the volume setup will error
-                                          unless it is marked optional. Paths must
-                                          be relative and may not contain the '..'
-                                          path or start with '..'.
-                                        type: array
-                                        items:
-                                          description: Maps a string key to a path
-                                            within a volume.
-                                          type: object
-                                          required:
-                                          - key
-                                          - path
-                                          properties:
-                                            key:
-                                              description: The key to project.
-                                              type: string
-                                            mode:
-                                              description: 'Optional: mode bits to
-                                                use on this file, must be a value
-                                                between 0 and 0777. If not specified,
-                                                the volume defaultMode will be used.
-                                                This might be in conflict with other
-                                                options that affect the file mode,
-                                                like fsGroup, and the result can be
-                                                other mode bits set.'
-                                              type: integer
-                                              format: int32
-                                            path:
-                                              description: The relative path of the
-                                                file to map the key to. May not be
-                                                an absolute path. May not contain
-                                                the path element '..'. May not start
-                                                with the string '..'.
-                                              type: string
-                                      name:
-                                        description: 'Name of the referent. More info:
-                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                          TODO: Add other useful fields. apiVersion,
-                                          kind, uid?'
-                                        type: string
-                                      optional:
-                                        description: Specify whether the Secret or
-                                          its key must be defined
-                                        type: boolean
-                                  serviceAccountToken:
-                                    description: information about the serviceAccountToken
-                                      data to project
-                                    type: object
-                                    required:
-                                    - path
-                                    properties:
-                                      audience:
-                                        description: Audience is the intended audience
-                                          of the token. A recipient of a token must
-                                          identify itself with an identifier specified
-                                          in the audience of the token, and otherwise
-                                          should reject the token. The audience defaults
-                                          to the identifier of the apiserver.
-                                        type: string
-                                      expirationSeconds:
-                                        description: ExpirationSeconds is the requested
-                                          duration of validity of the service account
-                                          token. As the token approaches expiration,
-                                          the kubelet volume plugin will proactively
-                                          rotate the service account token. The kubelet
-                                          will start trying to rotate the token if
-                                          the token is older than 80 percent of its
-                                          time to live or if the token is older than
-                                          24 hours.Defaults to 1 hour and must be
-                                          at least 10 minutes.
-                                        type: integer
-                                        format: int64
-                                      path:
-                                        description: Path is the path relative to
-                                          the mount point of the file to project the
-                                          token into.
-                                        type: string
-                        quobyte:
-                          description: Quobyte represents a Quobyte mount on the host
-                            that shares a pod's lifetime
-                          type: object
-                          required:
-                          - registry
-                          - volume
-                          properties:
-                            group:
-                              description: Group to map volume access to Default is
-                                no group
-                              type: string
-                            readOnly:
-                              description: ReadOnly here will force the Quobyte volume
-                                to be mounted with read-only permissions. Defaults
-                                to false.
-                              type: boolean
-                            registry:
-                              description: Registry represents a single or multiple
-                                Quobyte Registry services specified as a string as
-                                host:port pair (multiple entries are separated with
-                                commas) which acts as the central registry for volumes
-                              type: string
-                            tenant:
-                              description: Tenant owning the given Quobyte volume
-                                in the Backend Used with dynamically provisioned Quobyte
-                                volumes, value is set by the plugin
-                              type: string
-                            user:
-                              description: User to map volume access to Defaults to
-                                serivceaccount user
-                              type: string
-                            volume:
-                              description: Volume is a string that references an already
-                                created Quobyte volume by name.
-                              type: string
-                        rbd:
-                          description: 'RBD represents a Rados Block Device mount
-                            on the host that shares a pod''s lifetime. More info:
-                            https://examples.k8s.io/volumes/rbd/README.md'
-                          type: object
-                          required:
-                          - image
-                          - monitors
-                          properties:
-                            fsType:
-                              description: 'Filesystem type of the volume that you
-                                want to mount. Tip: Ensure that the filesystem type
-                                is supported by the host operating system. Examples:
-                                "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4"
-                                if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd
-                                TODO: how do we prevent errors in the filesystem from
-                                compromising the machine'
-                              type: string
-                            image:
-                              description: 'The rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: string
-                            keyring:
-                              description: 'Keyring is the path to key ring for RBDUser.
-                                Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: string
-                            monitors:
-                              description: 'A collection of Ceph monitors. More info:
-                                https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: array
-                              items:
-                                type: string
-                            pool:
-                              description: 'The rados pool name. Default is rbd. More
-                                info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: string
-                            readOnly:
-                              description: 'ReadOnly here will force the ReadOnly
-                                setting in VolumeMounts. Defaults to false. More info:
-                                https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: boolean
-                            secretRef:
-                              description: 'SecretRef is name of the authentication
-                                secret for RBDUser. If provided overrides keyring.
-                                Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            user:
-                              description: 'The rados user name. Default is admin.
-                                More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it'
-                              type: string
-                        scaleIO:
-                          description: ScaleIO represents a ScaleIO persistent volume
-                            attached and mounted on Kubernetes nodes.
-                          type: object
-                          required:
-                          - gateway
-                          - secretRef
-                          - system
-                          properties:
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Default is "xfs".
-                              type: string
-                            gateway:
-                              description: The host address of the ScaleIO API Gateway.
-                              type: string
-                            protectionDomain:
-                              description: The name of the ScaleIO Protection Domain
-                                for the configured storage.
-                              type: string
-                            readOnly:
-                              description: Defaults to false (read/write). ReadOnly
-                                here will force the ReadOnly setting in VolumeMounts.
-                              type: boolean
-                            secretRef:
-                              description: SecretRef references to the secret for
-                                ScaleIO user and other sensitive information. If this
-                                is not provided, Login operation will fail.
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            sslEnabled:
-                              description: Flag to enable/disable SSL communication
-                                with Gateway, default false
-                              type: boolean
-                            storageMode:
-                              description: Indicates whether the storage for a volume
-                                should be ThickProvisioned or ThinProvisioned. Default
-                                is ThinProvisioned.
-                              type: string
-                            storagePool:
-                              description: The ScaleIO Storage Pool associated with
-                                the protection domain.
-                              type: string
-                            system:
-                              description: The name of the storage system as configured
-                                in ScaleIO.
-                              type: string
-                            volumeName:
-                              description: The name of a volume already created in
-                                the ScaleIO system that is associated with this volume
-                                source.
-                              type: string
-                        secret:
-                          description: 'Secret represents a secret that should populate
-                            this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret'
-                          type: object
-                          properties:
-                            defaultMode:
-                              description: 'Optional: mode bits to use on created
-                                files by default. Must be a value between 0 and 0777.
-                                Defaults to 0644. Directories within the path are
-                                not affected by this setting. This might be in conflict
-                                with other options that affect the file mode, like
-                                fsGroup, and the result can be other mode bits set.'
-                              type: integer
-                              format: int32
-                            items:
-                              description: If unspecified, each key-value pair in
-                                the Data field of the referenced Secret will be projected
-                                into the volume as a file whose name is the key and
-                                content is the value. If specified, the listed keys
-                                will be projected into the specified paths, and unlisted
-                                keys will not be present. If a key is specified which
-                                is not present in the Secret, the volume setup will
-                                error unless it is marked optional. Paths must be
-                                relative and may not contain the '..' path or start
-                                with '..'.
-                              type: array
-                              items:
-                                description: Maps a string key to a path within a
-                                  volume.
-                                type: object
-                                required:
-                                - key
-                                - path
-                                properties:
-                                  key:
-                                    description: The key to project.
-                                    type: string
-                                  mode:
-                                    description: 'Optional: mode bits to use on this
-                                      file, must be a value between 0 and 0777. If
-                                      not specified, the volume defaultMode will be
-                                      used. This might be in conflict with other options
-                                      that affect the file mode, like fsGroup, and
-                                      the result can be other mode bits set.'
-                                    type: integer
-                                    format: int32
-                                  path:
-                                    description: The relative path of the file to
-                                      map the key to. May not be an absolute path.
-                                      May not contain the path element '..'. May not
-                                      start with the string '..'.
-                                    type: string
-                            optional:
-                              description: Specify whether the Secret or its keys
-                                must be defined
-                              type: boolean
-                            secretName:
-                              description: 'Name of the secret in the pod''s namespace
-                                to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret'
-                              type: string
-                        storageos:
-                          description: StorageOS represents a StorageOS volume attached
-                            and mounted on Kubernetes nodes.
-                          type: object
-                          properties:
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Implicitly inferred to be "ext4" if
-                                unspecified.
-                              type: string
-                            readOnly:
-                              description: Defaults to false (read/write). ReadOnly
-                                here will force the ReadOnly setting in VolumeMounts.
-                              type: boolean
-                            secretRef:
-                              description: SecretRef specifies the secret to use for
-                                obtaining the StorageOS API credentials.  If not specified,
-                                default values will be attempted.
-                              type: object
-                              properties:
-                                name:
-                                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
-                                    TODO: Add other useful fields. apiVersion, kind,
-                                    uid?'
-                                  type: string
-                            volumeName:
-                              description: VolumeName is the human-readable name of
-                                the StorageOS volume.  Volume names are only unique
-                                within a namespace.
-                              type: string
-                            volumeNamespace:
-                              description: VolumeNamespace specifies the scope of
-                                the volume within StorageOS.  If no namespace is specified
-                                then the Pod's namespace will be used.  This allows
-                                the Kubernetes name scoping to be mirrored within
-                                StorageOS for tighter integration. Set VolumeName
-                                to any name to override the default behaviour. Set
-                                to "default" if you are not using namespaces within
-                                StorageOS. Namespaces that do not pre-exist within
-                                StorageOS will be created.
-                              type: string
-                        vsphereVolume:
-                          description: VsphereVolume represents a vSphere volume attached
-                            and mounted on kubelets host machine
-                          type: object
-                          required:
-                          - volumePath
-                          properties:
-                            fsType:
-                              description: Filesystem type to mount. Must be a filesystem
-                                type supported by the host operating system. Ex. "ext4",
-                                "xfs", "ntfs". Implicitly inferred to be "ext4" if
-                                unspecified.
-                              type: string
-                            storagePolicyID:
-                              description: Storage Policy Based Management (SPBM)
-                                profile ID associated with the StoragePolicyName.
-                              type: string
-                            storagePolicyName:
-                              description: Storage Policy Based Management (SPBM)
-                                profile name.
-                              type: string
-                            volumePath:
-                              description: Path that identifies vSphere volume vmdk
-                              type: string
-              installPlanApproval:
-                description: Approval is the user approval policy for an InstallPlan.
-                  It must be one of "Automatic" or "Manual".
-                type: string
-              name:
-                type: string
-              source:
-                type: string
-              sourceNamespace:
-                type: string
-              startingCSV:
-                type: string
-          status:
-            type: object
-            required:
-            - lastUpdated
-            properties:
-              catalogHealth:
-                description: CatalogHealth contains the Subscription's view of its
-                  relevant CatalogSources' status. It is used to determine SubscriptionStatusConditions
-                  related to CatalogSources.
-                type: array
-                items:
-                  description: SubscriptionCatalogHealth describes the health of a
-                    CatalogSource the Subscription knows about.
-                  type: object
-                  required:
-                  - catalogSourceRef
-                  - healthy
-                  - lastUpdated
-                  properties:
-                    catalogSourceRef:
-                      description: CatalogSourceRef is a reference to a CatalogSource.
-                      type: object
-                      properties:
-                        apiVersion:
-                          description: API version of the referent.
-                          type: string
-                        fieldPath:
-                          description: 'If referring to a piece of an object instead
-                            of an entire object, this string should contain a valid
-                            JSON/Go field access statement, such as desiredState.manifest.containers[2].
-                            For example, if the object reference is to a container
-                            within a pod, this would take on a value like: "spec.containers{name}"
-                            (where "name" refers to the name of the container that
-                            triggered the event) or if no container name is specified
-                            "spec.containers[2]" (container with index 2 in this pod).
-                            This syntax is chosen only to have some well-defined way
-                            of referencing a part of an object. TODO: this design
-                            is not final and this field is subject to change in the
-                            future.'
-                          type: string
-                        kind:
-                          description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                          type: string
-                        name:
-                          description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                          type: string
-                        namespace:
-                          description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                          type: string
-                        resourceVersion:
-                          description: 'Specific resourceVersion to which this reference
-                            is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                          type: string
-                        uid:
-                          description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                          type: string
-                    healthy:
-                      description: Healthy is true if the CatalogSource is healthy;
-                        false otherwise.
-                      type: boolean
-                    lastUpdated:
-                      description: LastUpdated represents the last time that the CatalogSourceHealth
-                        changed
-                      type: string
-                      format: date-time
-              conditions:
-                description: Conditions is a list of the latest available observations
-                  about a Subscription's current state.
-                type: array
-                items:
-                  description: SubscriptionCondition represents the latest available
-                    observations of a Subscription's state.
-                  type: object
-                  required:
-                  - status
-                  - type
-                  properties:
-                    lastHeartbeatTime:
-                      description: LastHeartbeatTime is the last time we got an update
-                        on a given condition
-                      type: string
-                      format: date-time
-                    lastTransitionTime:
-                      description: LastTransitionTime is the last time the condition
-                        transit from one status to another
-                      type: string
-                      format: date-time
-                    message:
-                      description: Message is a human-readable message indicating
-                        details about last transition.
-                      type: string
-                    reason:
-                      description: Reason is a one-word CamelCase reason for the condition's
-                        last transition.
-                      type: string
-                    status:
-                      description: Status is the status of the condition, one of True,
-                        False, Unknown.
-                      type: string
-                    type:
-                      description: Type is the type of Subscription condition.
-                      type: string
-              currentCSV:
-                description: CurrentCSV is the CSV the Subscription is progressing
-                  to.
-                type: string
-              installPlanGeneration:
-                description: InstallPlanGeneration is the current generation of the
-                  installplan
-                type: integer
-              installPlanRef:
-                description: InstallPlanRef is a reference to the latest InstallPlan
-                  that contains the Subscription's current CSV.
-                type: object
-                properties:
-                  apiVersion:
-                    description: API version of the referent.
-                    type: string
-                  fieldPath:
-                    description: 'If referring to a piece of an object instead of
-                      an entire object, this string should contain a valid JSON/Go
-                      field access statement, such as desiredState.manifest.containers[2].
-                      For example, if the object reference is to a container within
-                      a pod, this would take on a value like: "spec.containers{name}"
-                      (where "name" refers to the name of the container that triggered
-                      the event) or if no container name is specified "spec.containers[2]"
-                      (container with index 2 in this pod). This syntax is chosen
-                      only to have some well-defined way of referencing a part of
-                      an object. TODO: this design is not final and this field is
-                      subject to change in the future.'
-                    type: string
-                  kind:
-                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-                    type: string
-                  name:
-                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
-                    type: string
-                  namespace:
-                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
-                    type: string
-                  resourceVersion:
-                    description: 'Specific resourceVersion to which this reference
-                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
-                    type: string
-                  uid:
-                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
-                    type: string
-              installedCSV:
-                description: InstalledCSV is the CSV currently installed by the Subscription.
-                type: string
-              installplan:
-                description: 'Install is a reference to the latest InstallPlan generated
-                  for the Subscription. DEPRECATED: InstallPlanRef'
-                type: object
-                required:
-                - apiVersion
-                - kind
-                - name
-                - uuid
-                properties:
-                  apiVersion:
-                    type: string
-                  kind:
-                    type: string
-                  name:
-                    type: string
-                  uuid:
-                    description: UID is a type that holds unique ID values, including
-                      UUIDs.  Because we don't ONLY use UUIDs, this is an alias to
-                      string.  Being a type captures intent and helps make sure that
-                      UIDs and names do not get conflated.
-                    type: string
-              lastUpdated:
-                description: LastUpdated represents the last time that the Subscription
-                  status was updated.
-                type: string
-                format: date-time
-              reason:
-                description: Reason is the reason the Subscription was transitioned
-                  to its current state.
-                type: string
-              state:
-                description: State represents the current state of the Subscription
-                type: string
-    served: true
-    storage: true
-    subresources:
-      status: {}
-
diff --git a/tools/k8s/devops/operators/olm-operator/olm.yaml b/tools/k8s/devops/operators/olm-operator/olm.yaml
deleted file mode 100644
index 5f3bc3af1..000000000
--- a/tools/k8s/devops/operators/olm-operator/olm.yaml
+++ /dev/null
@@ -1,347 +0,0 @@
----
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: olm
----
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: operators
----
-kind: ServiceAccount
-apiVersion: v1
-metadata:
-  name: olm-operator-serviceaccount
-  namespace: olm
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: system:controller:operator-lifecycle-manager
-rules:
-- apiGroups: ["*"]
-  resources: ["*"]
-  verbs: ["*"]
-- nonResourceURLs: ["*"]
-  verbs: ["*"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: olm-operator-binding-olm
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: system:controller:operator-lifecycle-manager
-subjects:
-- kind: ServiceAccount
-  name: olm-operator-serviceaccount
-  namespace: olm
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: olm-operator
-  namespace: olm
-  labels:
-    app: olm-operator
-spec:
-  strategy:
-    type: RollingUpdate
-  replicas: 1
-  selector:
-    matchLabels:
-      app: olm-operator
-  template:
-    metadata:
-      labels:
-        app: olm-operator
-    spec:
-      serviceAccountName: olm-operator-serviceaccount
-      containers:
-        - name: olm-operator
-          command:
-          - /bin/olm
-          args:
-          - --namespace
-          - $(OPERATOR_NAMESPACE)
-          - --writeStatusName
-          - ""
-          image: quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607
-          imagePullPolicy: IfNotPresent
-          ports:
-            - containerPort: 8080
-            - containerPort: 8081
-              name: metrics
-              protocol: TCP
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-          readinessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-          terminationMessagePolicy: FallbackToLogsOnError
-          env:
-        
-          - name: OPERATOR_NAMESPACE
-            valueFrom:
-              fieldRef:
-                fieldPath: metadata.namespace
-          - name: OPERATOR_NAME
-            value: olm-operator
-          resources:
-            requests:
-              cpu: 10m
-              memory: 160Mi
-          
-      
-      nodeSelector:
-        kubernetes.io/os: linux
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: catalog-operator
-  namespace: olm
-  labels:
-    app: catalog-operator
-spec:
-  strategy:
-    type: RollingUpdate
-  replicas: 1
-  selector:
-    matchLabels:
-      app: catalog-operator
-  template:
-    metadata:
-      labels:
-        app: catalog-operator
-    spec:
-      serviceAccountName: olm-operator-serviceaccount
-      containers:
-        - name: catalog-operator
-          command:
-          - /bin/catalog
-          args:
-          - '-namespace'
-          - olm
-          - -configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest
-          - -util-image
-          -  quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607
-          image: quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607
-          imagePullPolicy: IfNotPresent
-          ports:
-            - containerPort: 8080
-            - containerPort: 8081
-              name: metrics
-              protocol: TCP
-          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-          readinessProbe:
-            httpGet:
-              path: /healthz
-              port: 8080
-          terminationMessagePolicy: FallbackToLogsOnError
-          env:
-          
-          resources:
-            requests:
-              cpu: 10m
-              memory: 80Mi
-          
-      
-      nodeSelector:
-        kubernetes.io/os: linux
----
-kind: ClusterRole
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: aggregate-olm-edit
-  labels:
-    rbac.authorization.k8s.io/aggregate-to-admin: "true"
-    rbac.authorization.k8s.io/aggregate-to-edit: "true"
-rules:
-- apiGroups: ["operators.coreos.com"]
-  resources: ["subscriptions"]
-  verbs: ["create", "update", "patch", "delete"]
-- apiGroups: ["operators.coreos.com"]
-  resources: ["clusterserviceversions", "catalogsources", "installplans", "subscriptions"]
-  verbs: ["delete"]
----
-kind: ClusterRole
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: aggregate-olm-view
-  labels:
-    rbac.authorization.k8s.io/aggregate-to-admin: "true"
-    rbac.authorization.k8s.io/aggregate-to-edit: "true"
-    rbac.authorization.k8s.io/aggregate-to-view: "true"
-rules:
-- apiGroups: ["operators.coreos.com"]
-  resources: ["clusterserviceversions", "catalogsources", "installplans", "subscriptions", "operatorgroups"]
-  verbs: ["get", "list", "watch"]
-- apiGroups: ["packages.operators.coreos.com"]
-  resources: ["packagemanifests", "packagemanifests/icon"]
-  verbs: ["get", "list", "watch"]
----
-apiVersion: operators.coreos.com/v1
-kind: OperatorGroup
-metadata:
-  name: global-operators
-  namespace: operators
----
-apiVersion: operators.coreos.com/v1
-kind: OperatorGroup
-metadata:
-  name: olm-operators
-  namespace: olm
-spec:
-  targetNamespaces:
-    - olm
----
-apiVersion: operators.coreos.com/v1alpha1
-kind: ClusterServiceVersion
-metadata:
-  name: packageserver
-  namespace: olm
-  labels:
-    olm.version: 0.17.0
-spec:
-  displayName: Package Server
-  description: Represents an Operator package that is available from a given CatalogSource which will resolve to a ClusterServiceVersion.
-  minKubeVersion: 1.11.0
-  keywords: ['packagemanifests', 'olm', 'packages']
-  maintainers:
-  - name: Red Hat
-    email: openshift-operators@redhat.com
-  provider:
-    name: Red Hat
-  links:
-  - name: Package Server
-    url: https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server
-  installModes:
-  - type: OwnNamespace
-    supported: true
-  - type: SingleNamespace
-    supported: true
-  - type: MultiNamespace
-    supported: true
-  - type: AllNamespaces
-    supported: true
-  install:
-    strategy: deployment
-    spec:
-      clusterPermissions:
-      - serviceAccountName: olm-operator-serviceaccount
-        rules:
-        - apiGroups:
-            - authorization.k8s.io
-          resources:
-            - subjectaccessreviews
-          verbs:
-            - create
-            - get
-        - apiGroups:
-          - ""
-          resources:
-          - configmaps
-          verbs:
-          - get
-          - list
-          - watch
-        - apiGroups:
-          - "operators.coreos.com"
-          resources:
-          - catalogsources
-          verbs:
-          - get
-          - list
-          - watch
-        - apiGroups:
-          - "packages.operators.coreos.com"
-          resources:
-          - packagemanifests
-          verbs:
-          - get
-          - list
-      deployments:
-      - name: packageserver
-        spec:
-          strategy:
-            type: RollingUpdate
-          replicas: 2
-          selector:
-            matchLabels:
-              app: packageserver
-          template:
-            metadata:
-              labels:
-                app: packageserver
-            spec:
-              serviceAccountName: olm-operator-serviceaccount
-              nodeSelector:
-                kubernetes.io/os: linux
-              containers:
-              - name: packageserver
-                command:
-                - /bin/package-server
-                - -v=4
-                - --secure-port
-                - "5443"
-                - --global-namespace
-                - olm
-                image: quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607
-                imagePullPolicy: Always
-                ports:
-                - containerPort: 5443
-                livenessProbe:
-                  httpGet:
-                    scheme: HTTPS
-                    path: /healthz
-                    port: 5443
-                readinessProbe:
-                  httpGet:
-                    scheme: HTTPS
-                    path: /healthz
-                    port: 5443
-                terminationMessagePolicy: FallbackToLogsOnError
-                resources:
-                  requests:
-                    cpu: 10m
-                    memory: 50Mi
-                securityContext:
-                  runAsUser: 1000
-                volumeMounts:
-                - name: tmpfs
-                  mountPath: /tmp
-              volumes:
-              - name: tmpfs
-                emptyDir: {}
-  maturity: alpha
-  version: 0.17.0
-  apiservicedefinitions:
-    owned:
-    - group: packages.operators.coreos.com
-      version: v1
-      kind: PackageManifest
-      name: packagemanifests
-      displayName: PackageManifest
-      description: A PackageManifest is a resource generated from existing CatalogSources and their ConfigMaps
-      deploymentName: packageserver
-      containerPort: 5443
----
-apiVersion: operators.coreos.com/v1alpha1
-kind: CatalogSource
-metadata:
-  name: operatorhubio-catalog
-  namespace: olm
-spec:
-  sourceType: grpc
-  image: quay.io/operatorhubio/catalog:latest
-  displayName: Community Operators
-  publisher: OperatorHub.io
diff --git a/tools/scripts/devops b/tools/scripts/devops
deleted file mode 100755
index 0600f9e25..000000000
--- a/tools/scripts/devops
+++ /dev/null
@@ -1,161 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-_devops_subcommands="install uninstall completion install-completion status"
-_usage()
-{
-  echo 'usage ' >&2
-  echo ' '$(basename $0)' [-h] '$(echo $_devops_subcommands|sed 's/ /|/g') >&2
-  exit 0
-}
-
-_devops_completion()
-{
-  __devops_completion()
-  {
-    local _cur _prev _opts
-    COMPREPLY=()
-    _cur="${COMP_WORDS[COMP_CWORD]}"
-    _prev="${COMP_WORDS[COMP_CWORD-1]}"
-    _opts="-h --help install uninstall completion install-completion status"
-    case "${_prev}" in
-      -h|--help)
-        local base=${COMP_WORDS[COMP_CWORD-2]}
-        case "${base}" in
-          *)
-            return 0
-            ;;
-        esac
-        ;;
-      *)
-        if (( ${#COMP_WORDS[@]} > 2 )); then
-          local _options=$(eval "${COMP_WORDS[@]:0:$((${#COMP_WORDS[@]}-1))} options")
-          COMPREPLY=( $(compgen -W "${_options}" -- ${_cur}) )
-          return 0
-        fi
-        ;;
-    esac
-    COMPREPLY=($(compgen -W "${_opts}" -- ${_cur}))  
-    return 0
-  }
-  complete -F __devops_completion devops
-}
-
-_devops_install_completion()
-{
-  if [[ -d /usr/local/etc/bash_completion.d ]]; then
-    $0 completion > /usr/local/etc/bash_completion.d/$(basename $0)
-  fi
-}
-
-_devops_status()
-{
-  kubectl get ns -oname 2>/dev/null | grep mpi-operator 1>/dev/null 2>/dev/null
-  if (( $? == 0 )); then
-    echo '📦 installed'
-  else
-    echo '📦 not installed'
-  fi
-}
-
-_reverse() {
-  local _result=() _i=$(($# - 1)) _item _args=()
-  for _arg; do
-    _args+=("$_arg")
-  done
-  while (( $_i >= 0 )); do
-    _item=${_args[$_i]}
-    _result+=( $_item )
-    _i=$(($_i-1))
-  done
-  echo ${_result[@]}
-}
-
-_kubectl_action()
-{
-  local _array=() _dir _path=$1 _action=$2 _list=("operators")
-  if [[ $_action == delete ]]; then
-    _array=( $(_reverse ${_list[@]}) )
-  else
-    _array=( ${_list[@]} )
-  fi
-  for _dir in "${_array[@]}"; do
-    echo kubectl -k $_path/$_dir $_action
-    kubectl -k $_path/$_dir $_action
-  done
-}
-
-_main()
-{
-  local _kustomize_path=${0%%/scripts*}/k8s/devops
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-          _usage
-          exit 0
-          ;;
-      *)
-          echo 'unknown option '$1
-          shift
-          exit 1
-          ;;
-    esac
-  done
-  case "$#" in
-    0) 
-      _usage
-      ;;
-    1)
-      case "$1" in 
-        completion)
-          type _devops_completion | sed '1,3d;$d'
-          ;;
-        install)
-          _kubectl_action $_kustomize_path apply
-          ;;
-        install-completion)
-          _devops_install_completion
-          ;;
-        options)
-          echo '-h --help '$(echo $_devops_subcommands)' options '
-          return 0
-          ;;
-        status)
-          _devops_status
-          ;;
-        uninstall)
-          _kubectl_action $_kustomize_path delete
-          ;;
-        *)
-          echo 'unknown option '$1 >&2
-          _usage
-          ;;
-      esac
-      ;;
-    *) 
-      shift
-      echo 'unknown options '$@ >&2
-      ;;
-  esac
-}
-
-_main $@
-        
-
diff --git a/tools/scripts/mlops b/tools/scripts/mlops
deleted file mode 100755
index ee99d7ce1..000000000
--- a/tools/scripts/mlops
+++ /dev/null
@@ -1,315 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-_mlops_targets="$(echo $(for i in $(find * -type d -name k8s);do name=${i##*tensorflow/}; name=${name%/training*}; echo $name; done))"
-_mlops_targets_case=$(echo "$_mlops_targets"|sed 's/ /|/g')
-_mlops_subcommands="deploy exec logs status undeploy"
-_mlops_subcommands_case=$(echo "$_mlops_subcommands"|sed 's/ /|/g')
-_mlops_subsubcommands="single-node multi-node"
-_mlops_subsubcommands_case=$(echo "$_mlops_subsubcommands"|sed 's/ /|/g')
-_dry_run=false
-TARGETS='-h --help --dry-run options completion install-completion '"$_mlops_targets"
-
-_usage()
-{
-  echo 'usage ' >&2
-  echo ' '$(basename $0)' '$(echo $TARGETS|sed 's/ /|/g') >&2
-  exit 0
-}
-
-_mlops_completion()
-{
-  __mlops_completion()
-  {
-    local _cur _prev _opts
-    COMPREPLY=()
-    _cur="${COMP_WORDS[COMP_CWORD]}"
-    _prev="${COMP_WORDS[COMP_CWORD-1]}"
-    _opts="__TARGETS__"
-    case "${_prev}" in
-      completion|install-completion)
-        return 0
-        ;;
-      $_mlops_targets_case)
-        local _options="$_mlops_subcommands options"
-        COMPREPLY=( $(compgen -W "${_options}" -- ${_cur}) )
-        return 0
-        ;;
-      $_mlops_subcommands_case)
-        local _options="$_mlops_subsubcommands options"
-        COMPREPLY=( $(compgen -W "${_options}" -- ${_cur}) )
-        return 0
-        ;;
-      $_mlops_subsubcommands_case)
-        return 0
-        ;;
-      --dry-run)
-        local _options="$_mlops_targets options"
-        COMPREPLY=( $(compgen -W "${_options}" -- ${_cur}) )
-        return 0
-        ;;
-      -h|--help)
-        local base=${COMP_WORDS[COMP_CWORD-2]}
-        case "${base}" in
-          *)
-            return 0
-            ;;
-        esac
-        ;;
-      *)
-        if (( ${#COMP_WORDS[@]} > 2 )); then
-          local _options=$(eval "${COMP_WORDS[@]:0:$((${#COMP_WORDS[@]}-1))} options")
-          COMPREPLY=( $(compgen -W "${_options}" -- ${_cur}) )
-          return 0
-        fi
-        ;;
-    esac
-    COMPREPLY=($(compgen -W "${_opts}" -- ${_cur}))  
-    return 0
-  }
-  complete -F __mlops_completion mlops
-}
-
-_mlops_install_completion()
-{
-  if [[ -d /usr/local/etc/bash_completion.d ]]; then
-    $0 completion > /usr/local/etc/bash_completion.d/$(basename $0)
-  fi
-}
-
-_mlops_exec()
-{
-  local _container _pod _mpijob _container
-  case "$1" in 
-    multi-node)
-      _mpijob=$(kubectl get pods -oname 2>/dev/null | grep launcher | cut -c5-)
-      if [[ -n $_mpijob ]]; then
-        _mlops_invoke kubectl exec $_mpijob -c mpi-launcher -it -- bash
-      fi
-      ;;
-    single-node)
-      _job=$(kubectl get pods -oname 2>/dev/null | grep training | cut -c5-)
-      if [[ -n $_job ]]; then
-        _mlops_invoke kubectl exec $_job -c single-node -it -- bash
-      fi
-      ;;
-    *)
-      echo 'invalid option '$1 >&2
-      exit 1
-      ;;
-  esac
-}
-
-_mlops_logs()
-{
-  local _mpijob
-  case "$1" in 
-    multi-node)
-      _mpijob=$(kubectl get pods -oname 2>/dev/null | grep launcher | cut -c5-)
-      if [[ -n $_mpijob ]]; then
-        _mlops_invoke kubectl logs -f $_mpijob -c mpi-launcher
-      fi
-      ;;
-    single-node)
-      _job=$(kubectl get pods -oname 2>/dev/null | grep training | cut -c5-)
-      if [[ -n $_job ]]; then
-        _mlops_invoke kubectl logs -f $_job 
-      fi
-      ;;
-    *)
-      echo 'invalid option '$1 >&2
-      exit 1
-      ;;
-  esac
-}
-
-_mlops_status()
-{
-  case "$1" in 
-    multi-node)
-      _count=$(kubectl get mpijobs.kubeflow.org 2>/dev/null|wc -l|sed -e 's/[[:space:]]*$//')
-      if (( $_count > 0 )); then
-        echo '📬 deployed'
-      else
-        echo '📭 not deployed'
-      fi
-      ;;
-    single-node)
-      kubectl get pods 2>/dev/null | grep 'resnet50v1-5-fp32-training' 1>/dev/null 2>/dev/null
-      if (( $? == 0 )); then
-        echo '📬 deployed'
-      else
-        echo '📭 not deployed'
-      fi
-      ;;
-    *)
-      echo 'invalid option '$1 >&2
-      exit 1
-      ;;
-  esac
-}
-
-_mlops_kubectl_action()
-{
-  local _path=$1 _action=$2
-  case "$_action" in
-    apply)
-      _mlops_invoke 'kubectl kustomize '$_path' | '\
-        $'sed \'s/runAsUser:.*"\([0-9]*\)"/runAsUser: \\1/g\' | '\
-        $'sed \'s/runAsGroup:.*"\([0-9]*\)"/runAsGroup: \\1/g\' | '\
-        $'sed \'s/fsGroup:.*"\([0-9]*\)"/fsGroup: \\1/g\' | '\
-        'kubectl apply -f -'
-      ;;
-    *)
-      _mlops_invoke kubectl -k $_path $_action
-      ;;
-  esac
-}
-
-_mlops_invoke()
-{
-  if [[ $_dry_run == true ]]; then
-    echo $@
-  else
-    eval $@
-  fi
-}
-
-_main()
-{
-  local _kustomize_path=''
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-          _usage
-          exit 0
-          ;;
-      --dry-run)
-          shift
-          _dry_run=true
-          ;;
-      *)
-          echo 'unknown option '$1
-          shift
-          exit 1
-          ;;
-    esac
-  done
-  case "$#" in
-    0) 
-      _usage
-      ;;
-    1)
-      case "$1" in 
-        completion)
-          declare -a _targets=( $TARGETS )
-          type _mlops_completion | sed '1,3d;$d' | eval sed 's/__TARGETS__/'"'${_targets[@]}'"'/g' | sed 's^\$_mlops_targets_case^'$_mlops_targets_case'^g' | sed 's^\$_mlops_targets^'"$_mlops_targets"'^g' | sed 's^\$_mlops_subcommands_case^'$_mlops_subcommands_case'^g' | sed 's^\$_mlops_subcommands^'"$_mlops_subcommands"'^g' | sed 's^\$_mlops_subsubcommands_case^'$_mlops_subsubcommands_case'^g' | sed 's^\$_mlops_subsubcommands^'"$_mlops_subsubcommands"'^g'
-          ;;
-        install-completion)
-          _mlops_install_completion
-          ;;
-        options)
-          echo $TARGETS' options '
-          return 0
-          ;;
-        $_mlops_targets_case)
-          echo 'missing subcommand for '$1 >&2
-          _usage
-          exit 1
-          ;;
-        *)
-          echo 'unknown option '$1 >&2
-          _usage
-          exit 1
-          ;;
-      esac
-      ;;
-    2)
-      case "$2" in 
-        status)
-          echo 'missing single-node|multi-node option ' >&2
-          exit 1
-          ;;
-        exec)
-          echo 'missing single-node|multi-node option ' >&2
-          exit 1
-          ;;
-        logs)
-          echo 'missing single-node|multi-node option ' >&2
-          exit 1
-          ;;
-        options)
-          echo 'deploy exec logs options status undeploy'
-          return 0
-          ;;
-        deploy|undeploy)
-          echo 'missing single-node|multi-node option ' >&2
-          exit 1
-          ;;
-        *)
-          echo 'unknown option '$2 >&2
-          _usage
-          exit 1
-          ;;
-      esac
-      ;;
-    3)
-      _kustomize_path=$(find * -type d -name k8s|grep -v common)/mlops
-      case "$3" in 
-        single-node|multi-node)
-          case "$2" in 
-            deploy)
-              _mlops_kubectl_action $_kustomize_path/$3 apply
-              ;;
-            undeploy)
-              _mlops_kubectl_action $_kustomize_path/$3 delete
-              ;;
-            status)
-              _mlops_status $3
-              ;;
-            exec)
-              _mlops_exec $3
-              ;;
-            logs)
-              _mlops_logs $3
-              ;;
-            options)
-              echo '-h --help '$(echo $_mlops_subsubcommands)' options '
-              return 0
-              ;;
-            *)
-              echo 'unknown option '$2 >&2
-              _usage
-              ;;
-          esac
-          ;;
-        *) 
-          shift
-          echo 'unknown option '$3 >&2
-          ;;
-      esac
-      ;;
-    *) 
-      shift
-      echo 'unknown options '$@ >&2
-      ;;
-  esac
-}
-
-_main $@
-        
-
diff --git a/tools/scripts/model-builder b/tools/scripts/model-builder
deleted file mode 100755
index 19b6190c3..000000000
--- a/tools/scripts/model-builder
+++ /dev/null
@@ -1,2353 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-_usage()
-{
-  echo ''
-  echo 'Usage: '
-  echo ' '$(basename $0)' ['"$(echo ${_g_options[@]}|sed 's/ /|/g')"']' '<subcommand>' '[<model>...]'
-  echo ''
-  echo 'where a <subcommand> is one of:'
-  echo ' '$(echo ${_g_commands[@]})
-  echo ''
-  echo 'and where a <model> is one of:'
-  echo ' '"${_all_models[@]}"
-  echo ''
-  echo "Examples:"
-  echo "# -- Initialize a new model specification --"
-  echo 'model-builder init-spec -f <framework> <modelname-precision-mode> <use_case>'
-  echo ''
-  echo "# -- List all frameworks --"
-  echo 'model-builder frameworks'
-  echo ''
-  echo "# -- List all models in the default framework ("${DEFAULT_FRAMEWORK}") --"
-  echo 'model-builder models'
-  echo ''
-  echo "# -- Create two model packages --"
-  echo 'model-builder package <modelname> <modelname>'
-  echo ''
-  echo "# -- Create all model packages for the given framework --"
-  echo 'model-builder package -f <framework>'
-  echo ''
-  echo "# -- Build all model images from the given framework --"
-  echo 'model-builder build -f <framework>'
-  echo ''
-  echo "# -- View the call to docker run when generating a model's Dockerfile --"
-  echo "# -- Adding --dry-run will show the docker call but not execute it --"
-  echo 'model-builder --dry-run generate-dockerfile <modelname>'
-  echo ''
-  echo "# -- Generate a model's documentation --"
-  echo 'model-builder generate-documentation <modelname>'
-  echo ''
-  echo "# -- Do {generate-documentation,package,generate-dockerfile,build} for several models --"
-  echo 'model-builder make <modelname> <modelname>'
-  echo ''
-  echo "# -- Generate a k8 model's deployment file --"
-  echo 'model-builder generate-deployment <modelname>'
-  echo ''
-  echo "# -- Run the test-suite for the subcommand generate-documentation with the framework tensorflow and models: resnet50-fp32-inference and resnet50-int8-inference. --"
-  echo '  -- {subcommand,framework,[model model]} --'
-  echo 'model-builder run-test-suite --command generate-documentation --framework tensorflow resnet50-fp32-inference resnet50-int8-inference'
-  exit 0
-}
-
-#
-# Called at exit to clean up temp files and show source of error if one has occured
-# Outputs caller and error
-#
-_model_builder._onexit()
-{
-  if [[ -f $_all_specs ]]; then
-    echo ${_tmp_files[@]} | xargs rm -rf
-  fi
-  if (( $# > 0 )) && [[ -n $1 ]] && (( $1 != 0 )); then
-    # error handling goes here
-    echo "Script exited with error $1"
-  fi
-}
-
-#
-# Checks if the imz-tf-tools docker image exists
-# Returns:
-#  0 success
-#  1 failure
-#
-_model_builder._check-for-imz-tf-tools()
-{
-  local _verbose=''
-  if (( $# > 0 )); then
-    case "$1" in
-      --verbose)
-         _verbose='true'
-         ;;
-      *)
-         ;;
-    esac
-  fi
-  [[ -n $_verbose ]] && echo '> Checking for imz-tf-tools:latest container' >&2
-  docker inspect --format '{{index .RepoTags 0 }}' imz-tf-tools:latest 2>/dev/null 1>/dev/null
-}
-
-#
-# Builds the imz-tf-tools container
-# Returns:
-#  0 success
-#  1 failure
-#
-_model_builder._build-imz-tf-tools()
-{
-  local _dir _quiet='' _proxy_build
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in 
-      -q|--quiet)
-         _quiet=$1
-         shift
-         ;;
-      *)
-         echo 'unknown argument '$1
-         exit 1
-         ;;
-    esac
-  done
-  _dir=$1
-  if ! [[ -d $_dir ]]; then
-    echo $_dir' is not a directory'
-    exit 1
-  fi
-  if [[ -n $http_proxy && -n $https_proxy && -n $no_proxy ]]; then
-    _proxy_build=' --build-arg HTTPS_PROXY='$https_proxy' --build-arg HTTP_PROXY='$http_proxy' --build-arg NO_PROXY='$no_proxy
-  fi
-  test -z $_quiet && echo '> Building imz-tf-tools docker image' >&2
-  cat <<DOCKERIGNORE >.dockerignore
-*
-!bashrc
-DOCKERIGNORE
-  pushd $_dir 2>/dev/null 1>/dev/null
-  docker build $_quiet $_proxy_build --tag imz-tf-tools:latest -f tools.Dockerfile . && echo '> Build successful' >&2
-  docker_exit_code=$?
-  rm -f .dockerignore
-  popd 2>/dev/null 1>/dev/null
-
-  if [[ $docker_exit_code != 0 ]]; then
-    echo "ERROR: Unable to build the imz-tf-tools container to run the assembler"
-    exit $docker_exit_code
-  fi
-}
-
-#
-# Called to fetch proxy vars
-# Outputs format that can be used when calling docker run
-#
-_model_builder._get-proxy-env-vars()
-{
-  local _proxy_run=''
-  if [[ -n $http_proxy && -n $https_proxy && -n $no_proxy ]]; then
-    _proxy_run=' -e HTTPS_PROXY='$https_proxy' -e HTTP_PROXY='$http_proxy' -e NO_PROXY='$no_proxy
-  fi
-  echo "$_proxy_run"
-}
-
-#
-# Called by _model_builder._init to create functions that hold memoized info from the specifications.
-# Called within xargs to extract specification info.
-# Xargs allows parallel processing for performance.
-#
-_model_builder._generator()
-{
-  local _framework=$1
-  eval $'__model_builder_'$_framework$'_models() { echo $(ls '$_spec_folder$'/'$_framework$' | grep -v base_spec.yml | sed \'s/_spec.yml//\'); }'
-  eval "_model_builder_"$_framework"_models() { echo $(__model_builder_"$_framework"_models); }"
-  eval "type _model_builder_"$_framework"_models" | sed '1d' > $_tmp_dir/'_model_builder_'$_framework'_models'
-}
-
-#
-# Sets global vars 
-#  _current_models set to all models in _current_framework array
-#  _selected_models set to empty array
-#  _remaining_models set to _current_models
-#  _show_framework_options set to true
-#
-_model_builder._set_current_models()
-{
-  local _i
-  _current_models=()
-  _selected_models=()
-  _remaining_models=()
-  _show_framework_options=true
-
-  _current_models+=(${_all_models["$_current_framework"]});
-
-  if (( ${#_current_models[@]} > 0 )); then
-    _model_builder._sort_unique _current_models
-    _remaining_models=( ${_current_models[@]} )
-  fi
-}
-
-#
-# Utility function
-# Remove an item from an array. 
-# Can't use ~= since it matches substrings: 
-#   unet-fp32-inference would match 3d-unet-fp32-inference 
-#
-_model_builder._remove_item()
-{
-  local _target=$2 
-  declare -n _array=$1 
-  for i in "${!_array[@]}"; do
-    if [[ ${_array[i]} == $_target ]]; then
-      unset '_array[i]'
-    fi
-  done
-}
-
-#
-# Utility function
-# Rebuilds array to be sorted with unique entries
-#
-_model_builder._sort_unique()
-{
-  declare -n _array=$1 
-  _array=($(printf "%s\n" "${_array[@]}" | sort -u))
-}
-
-#
-# Called at start to initialize global vars from the specifications
-#
-_model_builder._init()
-{
-  local _asm_images _file _generator _i _ii _parallel _release=$1 _spec_folder _xdg_config_home _platform
-  _xdg_config_home=${XDG_CONFIG_HOME:-$HOME/.config}
-  declare -g _spec_home=${_xdg_config_home}/$(basename $0)/specs
-
-  trap '_model_builder._onexit $? $LINENO' EXIT
-
-  declare -g -r MODEL_DIR=${MODEL_DIR-$PWD}
-  # Tag to use for the 'versioned' images
-  declare -g -r TENSORFLOW_TAG=${TENSORFLOW_TAG:=latest}
-  # Image name to use for the 'versioned' images
-  declare -g -r TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE:=intel/intel-optimized-tensorflow}
-  # Local container repo used for building images
-  declare -g -r LOCAL_REPO=${LOCAL_REPO-model-zoo}
-  declare -g -r TAG_PREFIX=${TENSORFLOW_TAG}
-  # If no framework folder is specified, use tensorflow
-  declare -g -r DEFAULT_FRAMEWORK="tensorflow"
-  declare -g _current_framework=${DEFAULT_FRAMEWORK}
-
-  _model_builder._root 1>/dev/null
-  declare -g -r _g_root=$(_model_builder._root)
-  declare -g -r _g_options=("-h" "--help" "--verbose" "--dry-run")
-  declare -g _g_commands=($(echo $(cat $0|grep '^_model_builder\.[^_]\([a-zA-Z0-9_]*\)[ \t]*('|sed 's/_model_builder\.\([a-zA-Z0-9_]*\)[ \t]*()/\1/'|tr '_' '-'|sort)))
-  declare -g _g_commands_case_statement="$(_model_builder._join '|' ${_g_commands[@]})"
-
-  declare -g -A _all_models
-  declare -g -a _current_models _selected_models _remaining_models
-  declare -g -a _all_commands _current_commands _remaining_commands
-  declare -g -a _all_frameworks _remaining_frameworks
-  declare -g -a _release_groups
-  declare -g -a _all_specs _tmp_files
-  declare -g _show_framework_options=true
-
-  _model_builder._root 1>/dev/null
-
-  _spec_folder=$(_model_builder._root)/docker/specs
-  _spec_dir_mounted='./specs'
-
-  # add platforms other than ubuntu:
-  _platform='centos'
-  if [[ ${TAG_PREFIX} == *"${_platform}"* ]] || [[ ${TAG_PREFIX} == *"spr"* ]] ; then
-    _spec_folder=${_spec_folder}/${_platform}
-    _spec_home=${_spec_home}/${_platform}
-    _spec_dir_mounted=${_spec_dir_mounted}/${_platform}
-  fi
-
-  declare -g _spec_dir_mounted=${_spec_dir_mounted}
-
-  if [[ ! -d $_spec_folder ]]; then
-    echo "Error: The spec folder does not exist ($_spec_folder)."
-    exit 1
-  fi
-  if [[ ! -d $_spec_home ]]; then
-    mkdir -p $_spec_home && cp -r $_spec_folder/k8s $_spec_home && rm -f $_spec_home/k8s/base_spec.yml
-    if (( $? != 0 )); then
-      echo "Unable to create ${_spec_home}." >&2
-      exit 1
-    fi
-  fi
-
-  if (( ${#_all_frameworks[@]} == 0 )) && (( ${#_all_models[*]} == 0 )); then
-    _all_specs=$(mktemp /tmp/$(basename $0)_specs.XXXXXXXXX)
-    _tmp_files+=($_all_specs)
-
-    cat $(find ${_spec_folder} -name '*.yml') > $_all_specs
-
-    _all_commands=(generate-deployment generate-documentation package generate-dockerfile build)
-    eval $'__model_builder_frameworks_function() { [[ -d ${_spec_folder} ]] && echo $(ls $_spec_folder); }'
-
-    eval "_model_builder_frameworks_function() { echo $(__model_builder_frameworks_function); }"
-    _all_frameworks+=( $(_model_builder_frameworks_function) )
-
-    _generator=$(mktemp /tmp/model-builder._generator.XXXXXXXXX)
-    _tmp_files+=($_generator)
-    _tmp_dir=$(mktemp -d /tmp/model-builder._generator_files.XXXXXXXXX)
-    _tmp_files+=($_tmp_dir)
-    echo '#!/usr/bin/env bash' > $_generator
-    echo '_tmp_dir='$_tmp_dir >> $_generator
-    echo '_spec_folder='$_spec_folder >> $_generator
-    type _model_builder._generator | sed '1d' >> $_generator
-    echo '_model_builder._generator $1' >> $_generator
-    chmod +x $_generator
-    _parallel=${#_all_frameworks[@]}
-    echo ${_all_frameworks[@]} | xargs -n1 -P${#_all_frameworks[@]} $_generator
-    for _i in "${_all_frameworks[@]}"; do
-      _file=$_tmp_dir'/_model_builder_'$_i'_models'
-      eval "source $_file"
-      _ii="$(eval $'\"_model_builder_'$_i$'_models\"')"
-      _all_models["$_i"]=$_ii
-    done
-
-    _model_builder._set_current_models
-  fi
-}
-
-#
-# Utility function
-# Joins an array with the first arg being the delimiter 
-#
-_model_builder._join() 
-{
-  local IFS="$1"
-  shift
-  echo "$*"
-}
-
-#
-# Outputs the root directory of the model-builder repo
-#
-_model_builder._root()
-{
-  local _root=$(dirname $(dirname $0))
-  if [[ -d $_root/docker/specs ]]; then
-    echo $_root
-  elif [[ -d $PWD/tools/docker/specs ]]; then
-    echo $PWD/tools
-  else
-    echo $(basename $0)$' isn\'t in tools/scripts and/or the working directory isn\'t at the root of the model repository.' >&2
-    exit 1
-  fi
-}
-
-#
-# Outputs completion logic for a subcommand that only has cli options (-o or --option)
-#
-_model_builder._subcommand()
-{
-  local _arg _cur _i=2 _options _prev _subcommand=$1
-  case "$_subcommand" in
-    *)
-      while (( $_i < $COMP_CWORD )); do
-        local s="${COMP_WORDS[_i]}"
-        case "$s" in
-          -*) 
-            ;;
-        esac
-        (( _i++ ))
-      done
-      _cur="${COMP_WORDS[COMP_CWORD]}"
-      _prev="${COMP_WORDS[COMP_CWORD-1]}"
-      if (( $_i == $COMP_CWORD )); then
-        _options='-h --help'
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      ;;
-  esac
-}
-
-#
-# Outputs completion logic for subcommands that have
-#   (-f|--framework) <framework>
-#
-_model_builder._subcommand_show_frameworks()
-{
-  local _arg _cur _i=2 _options _prev _subcommand=$1
-  case "$_subcommand" in
-    images|models|packages)
-      while (( $_i < $COMP_CWORD )); do
-        local s="${COMP_WORDS[_i]}"
-        case "$s" in
-          -f|--framework)
-            if [[ $_first_framework == true ]]; then
-              _current_framework=''
-              _first_framework=false
-            fi
-            ;;
-          -*)
-            ;;
-          *)
-            if [[ "${_remaining_frameworks[@]}" =~ $s ]]; then
-              _current_framework=$s
-              _model_builder._remove_item _remaining_frameworks $s
-              _model_builder._set_current_models
-              _show_framework_options=false
-            fi
-            ;;
-        esac
-        (( _i++ ))
-      done
-      _cur="${COMP_WORDS[COMP_CWORD]}"
-      _prev="${COMP_WORDS[COMP_CWORD-1]}"
-      if [[ $_prev == --framework ]] || [[ $_prev == -f ]]; then
-        _options="$(echo ${_remaining_frameworks[@]})"
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      if (( $_i == $COMP_CWORD )); then
-        if [[ $_show_framework_options == true ]]; then
-          _options='-h --help -f --framework '
-        else
-          _options='-h --help '
-        fi
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      ;;
-  esac
-}
-
-#
-# Outputs completion logic for subcommands that have 
-#   (-f|--framework) <framework> models
-#
-_model_builder._subcommand_show_frameworks_and_models()
-{
-  local _arg _cur _i=2 _options _prev _subcommand=$1
-  case "$_subcommand" in
-    build|generate-deployment|generate-dockerfile|generate-documentation|make|package)
-      while (( $_i < $COMP_CWORD )); do
-        local s="${COMP_WORDS[_i]}"
-        case "$s" in
-          -f|--framework)
-            if [[ $_first_framework_group == true ]]; then
-              _current_framework=''
-              _first_framework=false
-            fi
-            ;;
-          -*) 
-            ;;
-          *)
-            if [[ "${_remaining_frameworks[@]}" =~ $s ]]; then
-              _current_framework=$s
-              _model_builder._remove_item _remaining_frameworks $s
-              _model_builder._set_current_models
-              _show_framework_options=false
-            fi
-            if [[ "${_remaining_models[@]}" =~ $s ]] && ! [[ "${_selected_models[@]}" =~ $s ]]; then
-              _show_framework_options=false
-              _selected_models+=( $s )
-              _model_builder._remove_item _remaining_models $s
-            fi
-            ;;
-        esac
-        (( _i++ ))
-      done
-      _cur="${COMP_WORDS[COMP_CWORD]}"
-      _prev="${COMP_WORDS[COMP_CWORD-1]}"
-      if [[ $_prev == --framework ]] || [[ $_prev == -f ]]; then
-        _options="$(echo ${_remaining_frameworks[@]})"
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      if (( $_i == $COMP_CWORD )); then
-        if [[ $_show_framework_options == true ]]; then
-          _options='-h --help -f --framework '"${_remaining_models[@]}"
-        else
-          _options='-h --help '"${_remaining_models[@]}"
-        fi
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      ;;
-  esac
-}
-
-#
-# Outputs completion logic for subcommands that have 
-#   (-c|--command) <command> (-f|--framework) <framework> models
-#
-_model_builder._subcommand_show_commands_frameworks_and_models()
-{
-  local _arg _cur _i=2 _options _prev _subcommand=$1
-  case "$_subcommand" in
-    run-test-suite)
-      while (( $_i < $COMP_CWORD )); do
-        local s="${COMP_WORDS[_i]}"
-        case "$s" in
-          -c|--command)
-            if [[ $_first_command == true ]]; then
-              _current_commands=()
-              _first_command=false
-            fi
-            ;;
-          -f|--framework)
-            if [[ $_first_framework == true ]]; then
-              _current_framework=''
-              _first_framework=false
-              _show_command_options=false
-            fi
-            ;;
-          -*)
-            if [[ "${_remaining_commands[@]}" =~ $s ]]; then
-              _current_commands+=( $s )
-              _model_builder._remove_item _remaining_commands $s
-              if (( ${#_remaining_commands[@]} == 0 )); then
-                _show_command_options=false
-              fi
-            fi
-            ;;
-          *)
-            if [[ "${_remaining_commands[@]}" =~ $s ]]; then
-              _current_commands+=( $s )
-              _model_builder._remove_item _remaining_commands $s
-              if (( ${#_remaining_commands[@]} == 0 )); then
-                _show_command_options=false
-              fi
-            fi
-            if [[ "${_remaining_frameworks[@]}" =~ $s ]]; then
-              _current_framework=$s
-              _model_builder._remove_item _remaining_frameworks $s
-              _model_builder._set_current_models
-              _show_command_options=false
-              _show_framework_options=false
-            fi
-            if [[ "${_remaining_models[@]}" =~ $s ]] && ! [[ "${_selected_models[@]}" =~ $s ]]; then
-              _show_command_options=false
-              _show_framework_options=false
-              _selected_models+=( $s )
-              _model_builder._remove_item _remaining_models $s
-            fi
-            ;;
-        esac
-        (( _i++ ))
-      done
-      _cur="${COMP_WORDS[COMP_CWORD]}"
-      _prev="${COMP_WORDS[COMP_CWORD-1]}"
-      if [[ $_prev == --command ]] || [[ $_prev == -c ]]; then
-        _options="$(echo ${_remaining_commands[@]})"
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      if [[ $_prev == --framework ]] || [[ $_prev == -f ]]; then
-        _options="$(echo ${_remaining_frameworks[@]})"
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      if (( $_i == $COMP_CWORD )); then
-        if [[ $_show_command_options == true ]]; then 
-          _options='-h --help -c --command -f --framework '"${_remaining_models[@]}"
-        elif [[ $_show_framework_options == true ]]; then
-          _options='-h --help -f --framework '"${_remaining_models[@]}"
-        else
-          _options='-h --help '"${_remaining_models[@]}"
-        fi
-        COMPREPLY=($(compgen -W "$_options" -- "$_cur"))
-        return
-      fi
-      ;;
-  esac
-}
-
-#
-# Main entry point for completion logic
-#
-_model_builder._completion()
-{
-  local _i=1 _subcommand
-  _model_builder._completion_init
-
-  _model_builder._set_current_models
-  # find the subcommand
-  while (( $_i < $COMP_CWORD )); do
-    local s="${COMP_WORDS[_i]}"
-    case "$s" in
-      -f|--framework)
-          (( _i++ ))
-          ;;
-      -*) ;;
-      *)
-        _subcommand="$s"
-        break
-        ;;
-    esac
-    (( _i++ ))
-  done
-
-  if (( $_i == $COMP_CWORD )); then
-    local cur="${COMP_WORDS[COMP_CWORD]}"
-    COMPREPLY=($(compgen -W "$(_model_builder._get_options)" -- "$cur"))
-    return
-  fi
-
-  # we've completed the 'current' command and now need to call the next completion function
-  # subcommands have their own completion functions
-  case "$_subcommand" in
-    commands|completion|init-spec|install-completion|frameworks)
-      _model_builder._subcommand $_subcommand
-      return
-      ;;
-    build|make)
-      _current_framework=$DEFAULT_FRAMEWORK
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_frameworks_and_models $_subcommand
-      return
-      ;;
-    images)
-      _current_framework=$DEFAULT_FRAMEWORK
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_frameworks $_subcommand
-      return
-      ;;
-    generate-dockerfile|generate-documentation|package)
-      _current_framework=$DEFAULT_FRAMEWORK
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_frameworks_and_models $_subcommand
-      return
-      ;;
-    generate-deployment)
-      _current_framework=k8s
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_frameworks_and_models $_subcommand
-      return
-      ;;
-    models|packages)
-      _current_framework=$DEFAULT_FRAMEWORK
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_frameworks $_subcommand
-      return
-      ;;
-    run-test-suite)
-      _current_commands=(${_all_commands[@]})
-      _remaining_commands=(${_current_commands[@]})
-      _current_framework=$DEFAULT_FRAMEWORK
-      _remaining_frameworks=(${_all_frameworks[@]})
-      _model_builder._set_current_models
-      _model_builder._subcommand_show_commands_frameworks_and_models $_subcommand
-      return
-      ;;
-    *)
-      echo $_subcommand' huh?' >&2
-      exit 1
-      ;;
-  esac
-}
-
-#
-# Outputs the bash completion script
-#
-_model_builder.completion()
-{
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Emit the bash completion source\n'
-        exit 0
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  eval $'__model_builder_options() { echo ${_g_options[@]}\' \'${_g_commands[@]}; }'
-  eval "_model_builder._get_options() { echo $(__model_builder_options); }"
-  echo '_model_builder._completion_init()'
-  echo '{'
-  _model_builder._export '_all_models' _all_models
-  echo '  declare -g '$'DEFAULT_FRAMEWORK=('${DEFAULT_FRAMEWORK}$')'
-  echo '  declare -g '$'_all_frameworks=('${_all_frameworks[@]}$')'
-  echo '  declare -g '$'_current_framework='${DEFAULT_FRAMEWORK}$
-  echo '  declare -g _remaining_frameworks=(${_current_framework[@]})'
-  echo '  declare -g _first_framework=true'
-  echo '  declare -g _show_framework_options=true'
-  echo '  declare -g _g_commands=('${_g_commands[@]}')'
-  echo '  declare -g _g_commands_case_statement="'$'$(_model_builder._join \'|\' ${_g_commands[@]})'$'"'
-  echo '  declare -g _first_command=true'
-  echo '  declare -g _show_command_options=true'
-  echo '  declare -g -a _all_commands=('${_all_commands[@]}')'
-  echo '  declare -g -a _current_commands'
-  echo '  declare -g -a _remaining_commands'
-  echo '  declare -g -a _current_models'
-  echo '  declare -g -a _selected_models'
-  echo '  declare -g -a _remaining_models'
-  echo '}'
-  type _model_builder._get_options | sed '1d'
-  type _model_builder._set_current_models | sed '1d'
-  type _model_builder._remove_item | sed '1d'
-  type _model_builder._sort_unique | sed '1d'
-  type _model_builder._join | sed '1d'
-  type _model_builder._subcommand | sed '1d'
-  type _model_builder._subcommand_show_frameworks | sed '1d'
-  type _model_builder._subcommand_show_frameworks_and_models | sed '1d'
-  type _model_builder._subcommand_show_commands_frameworks_and_models | sed '1d'
-  type _model_builder._completion | sed '1d'
-  echo 'complete -o bashdefault -F _model_builder._completion model-builder'
-}
-
-#
-# Echos all args if --verbose || --dry-run
-# Evals all args if ! --dry-run
-#
-_model_builder._echo_command()
-{
-  if [[ $@ =~ --verbose ]] || [[ $@ =~ --dry_run ]]; then
-    echo $@
-  fi
-  if ! [[ $@ =~ --dry_run ]]; then
-    eval $@
-  fi
-}
-
-#
-# Entry point for 'model-builder init-spec ...'
-# Creates model from specification
-#
-_model_builder.init_spec()
-{
-  local _args=() _asm_new_spec _new_model _model_dir _model_dir_mount _use_case
-
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Creates a new specification based on the model arguments\n'
-        return 0
-        ;;
-      --verbose)
-        _args+=("$1")
-        ;;
-      -f|--framework)
-        shift
-        ;;
-      --dry-run)
-        _args+=("--dry_run")
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  # Used for auto a new spec with the list of package files for the specified model
-
-  # Check for the tf-tool container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-
-  _new_model=$1
-  _use_case=$2
-
-  echo "Auto-generating a new spec file for: $_new_model"
-
-  _model_dir_mount='-v '$MODEL_DIR':/tf/models'
-  _model_dir='--model_dir=models'
-
-  _model_download=""
-  if ! [[ -z $MODEL_URL ]]; then
-    _model_download="--model_download=${MODEL_URL}"
-  fi
-
-  if [[ ! -z ${_use_case} ]]; then
-    echo "Using use case '${_use_case}' if the model's directories don't already exist"
-    _use_case="--use_case=${_use_case}"
-  fi
-
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-
-  # Note: Run as user so that the spec file isn't owned by root
-  _asm_new_spec="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) $_model_dir_mount -v $(pwd):/tf -it imz-tf-tools python3 assembler.py "
-
-  _model_builder._echo_command ${_asm_new_spec} \
-      --arg http_proxy=${http_proxy} \
-      --arg https_proxy=${https_proxy} \
-      --arg no_proxy=${no_proxy} \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --generate_new_spec=${_new_model} \
-      --framework=${_current_framework} \
-      ${_use_case} \
-      $_model_download \
-      $_model_dir \
-      ${_args[@]}
-  popd 1>/dev/null 2>/dev/null
-}
-
-_model_builder_get_images_from_log()
-{
-  # Checks the log file for the list of 'IMAGES_BUILT=' which should be
-  # formatted as a comma separated list. Returns that comma separated value
-  log_path=$1
-
-  if [[ -f ${log_path} ]]; then
-    # Find the last match for the list of images built in the output file
-    images_built=$(grep '^IMAGES_BUILT=' ${log_path} | tail -1)
-    # Strip off the "IMAGES_BUILT=" from that line to return
-    echo $(echo $images_built | sed 's/^IMAGES_BUILT=//')
-  else
-    echo ""
-  fi
-}
-
-#
-# Entry point for 'model-builder build ...'
-# Builds one or more images from models
-# Calls assembler.py
-#
-_model_builder.build()
-{
-  local _abs_model_dir='' _args=() _asm_images _count _dry_run=false _log_image_lists=() _model_dir='' 
-  local _model_dir_mount='' _model_package_folder _output_dir='' _release_args=() _release_group 
-  local _temp_model_package_dir _verbose='' 
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Builds the images for the specified models (default=*) in the specified framework (default='"${DEFAULT_FRAMEWORK}"').\n'
-        exit 0
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run") 
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --output_dir=*)
-        _output_dir=${1#*=}
-        _output_dir=${_output_dir##*/}
-        _args+=("$1") 
-        ;;
-      -f|--framework)
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-
-  # Define default values for the variables
-  if [ -z "${MODEL_PACKAGE_DIR}" ]; then
-    MODEL_PACKAGE_DIR=../output
-  fi
-
-  if [ -z "${MODEL_WORKSPACE}" ]; then
-    MODEL_WORKSPACE=/workspace
-  fi
-  
-  # Check for the tf-tool container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  
-  # Verify that the model package directory exists
-  if [ ! -d ${MODEL_PACKAGE_DIR} ]; then
-    echo "The model package directory was not found at: ${MODEL_PACKAGE_DIR}"
-    echo "Model packages must be built before building containers"
-    exit 1
-  fi 
-  
-  # Copy model package directory to this docker folder so that it's included in
-  # the imz-tf-tools volume mount and it'll be in the same build context (directory)
-  # that we are running from.
-  _model_package_folder="model_packages"
-  _temp_model_package_dir="$(pwd)/${_model_package_folder}/"
-  rm -rf ${_temp_model_package_dir}
-  mkdir -p ${_temp_model_package_dir}
-  _count=$(ls ${MODEL_PACKAGE_DIR}/*.tar.gz 2>/dev/null | wc -l)
-  if (( $_count > 0 )); then
-    cp ${MODEL_PACKAGE_DIR}/*.tar.gz ${_temp_model_package_dir}
-  fi
-
-  _asm_images="docker run --rm $(_model_builder._get-proxy-env-vars) -v $(pwd):/tf $_model_dir_mount -v /var/run/docker.sock:/var/run/docker.sock imz-tf-tools python3 assembler.py "
-
-  for _release_group in "${_release_groups[@]}"
-  do
-    temp_output_file=$(mktemp output.XXXXXXXXX)
-    _tmp_files+=($temp_output_file)
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=()
-        ;;
-    esac
-    _model_builder._echo_command ${_asm_images} \
-      --arg http_proxy=${http_proxy} \
-      --arg https_proxy=${https_proxy} \
-      --arg no_proxy=${no_proxy} \
-      --arg PACKAGE_DIR=${_model_package_folder} \
-      --arg MODEL_WORKSPACE=${MODEL_WORKSPACE} \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --repository ${LOCAL_REPO} \
-      --release ${_release_group} \
-      ${TEST_DIR:+"--run_tests_path" "$TEST_DIR"} \
-      --build_images \
-      ${_release_args[@]} \
-      ${_args[@]} 2>&1 | tee $temp_output_file
-
-    result=$?
-    if (( $result != 0 )); then
-      echo "Warning: A non-zero error code ($result) was returned when building the '$_release_group' images"
-    fi
-    _log_image_lists+=($(_model_builder_get_images_from_log $temp_output_file))
-    rm $temp_output_file
-  done
-
-  if [[ $_dry_run == false ]]; then
-    echo "--------------------------------------"
-  fi
-
-  # Print out the images that were just built, and append to the IMAGE_LIST_FILE
-  count=0
-  for _log_image_list in "${_log_image_lists[@]}"
-  do
-    for i in $(echo ${_log_image_list} | sed "s/,/ /g")
-    do
-      echo $i
-      count=$(($count+1))
-      if [[ ! -z ${IMAGE_LIST_FILE} ]]; then
-        echo $i >> ${IMAGE_LIST_FILE}
-      fi
-    done
-  done
-
-  if [[ $_dry_run == false ]] && [[ ${count} == 0 ]]; then
-    echo ""
-    echo "No containers were built for framework: ${_current_framework} "
-    echo "Check the log output above for errors and ensure that you have the proper --framework and build names specified."
-  fi
-
-  # Delete the temporary copy of the model package directory
-  rm -rf ${_temp_model_package_dir}
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Entry point for 'model-builder package ...'
-# Packages one or more models
-# Calls assembler.py
-#
-_model_builder.package()
-{
-  local _abs_model_dir='' _args=() _asm_dockerfiles _dry_run=false _model_dir='' _model_dir_mount='' _output_dir='' _release_args=() _framework _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Builds the model packages for the specified models (default=*) in the specified framework.\n'
-        return 0
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run")
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --output_dir=*)
-        _output_dir=${1#*=}
-        _output_dir=${_output_dir##*/}
-        _args+=("$1") 
-        ;;
-      -f|--framework)
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-
-  # Check for git branch/commit env vars (normally from Jenkins) for the wrapper package info.txt
-  _git_vars=""
-  if [[ ! -z "${GIT_BRANCH}" ]]; then
-    _git_vars=" --env GIT_BRANCH=${GIT_BRANCH}"
-  fi
-  if [[ ! -z "${GIT_COMMIT}" ]]; then
-    _git_vars="${_git_vars} --env GIT_COMMIT=${GIT_COMMIT}"
-  fi
-  
-  # Note: Run as user so that the dockerfiles have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) ${_git_vars} -v $(pwd):/tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  for _release_group in "${_release_groups[@]}"
-  do
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=()
-        ;;
-    esac
-    # Call the assembler to build model packages
-    _model_builder._echo_command ${_asm_dockerfiles} \
-      --release $_release_group \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --build_packages \
-      ${_release_args[@]} \
-      ${_args[@]}
-  done
-
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Entry point for 'model-builder generate-deployment ...'
-# Generates one or more yaml files from specified k8 packages
-#
-_model_builder.generate_deployment()
-{
-  local _abs_model_dir _args=() _asm_dockerfiles _count _dry_run=false _model_dir='' _model_dir_mount='' _output_dir='' _release_args=() _framework _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Generates k8 yaml files for the specified model in the k8 framework.\n'
-        return 0
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run")
-        ;;
-      -f|--framework)
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  
-  # Note: Run as user so that the generated deployment(s) have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) -e MODEL_BUILDER_SPEC_HOME=/specs $(_model_builder._get-proxy-env-vars) -v ${_spec_home}:/specs -v $(pwd):/tf -w /tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  for _release_group in "${_release_groups[@]}"
-  do
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=()
-        ;;
-    esac
-    # Call the assembler to generate deployments
-    _model_builder._echo_command ${_asm_dockerfiles} \
-      --release $_release_group \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --generate_deployments \
-      ${_release_args[@]} \
-      ${_args[@]}
-  done
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Entry point for 'model-builder generate-dockerfile ...'
-# Generates one or more dockerfiles from specified models
-# Calls assembler.py
-#
-_model_builder.generate_dockerfile()
-{
-  local _abs_model_dir _args=() _asm_dockerfiles _count _dry_run=false _model_dir='' _model_dir_mount='' _output_dir='' _release_args=() _framework _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Builds the dockerfiles for the specified models (default=*) in the specified framework.\n'
-        return 0
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run")
-        ;;
-      -f|--framework)
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  
-  # Note: Run as user so that the dockerfiles have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) -v $(pwd):/tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  for _release_group in "${_release_groups[@]}"
-  do
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=()
-        ;;
-    esac
-    # Call the assembler to construct dockerfiles
-    _model_builder._echo_command ${_asm_dockerfiles} \
-      --release $_release_group \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --construct_dockerfiles \
-      ${_release_args[@]} \
-      ${_args[@]}
-  done
-  popd 1>/dev/null 2>/dev/null
-  
-  if [[ $_dry_run == false ]]; then
-    if ! [[ -d dockerfiles ]]; then
-      echo $(pwd)/dockerfiles does not exist >&2
-      exit 1
-    fi
-  fi
-}
-
-#
-# Entry point for 'model-builder generate-documentation ...'
-# Creates one or more README.md's from specified models
-# Calls assembler.py
-#
-_model_builder.generate_documentation()
-{
-  local _abs_model_dir _args=() _asm_dockerfiles _count _model_dir='' _model_dir_mount='' _output_dir='' _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Builds a README.md file using the doc partials for the specified models (default=*) in the specified framework.\n'
-        return 0
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      --dry-run)
-        _args+=("--dry_run")
-        ;;
-      -f|--framework)
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-
-  # Note: Run as user so that the dockerfiles have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) -v $(pwd):/tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  for _release_group in "${_release_groups[@]}"
-  do
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=()
-        ;;
-    esac
-    # Call the assembler to generate documentation
-    _model_builder._echo_command ${_asm_dockerfiles} \
-      --release $_release_group \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      --generate_documentation \
-      ${_release_args[@]} \
-      ${_args[@]}
-  done
-  popd 1>/dev/null 2>/dev/null
-}
-
-_model_builder.commands()
-{
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'List all subcommands within model-builder\n'
-        exit 0
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo ${_g_commands[@]}
-}
-
-#
-# Entry point for 'model-builder images ...'
-#
-_model_builder.images()
-{
-  local _abs_model_dir _args=() _asm_dockerfiles _dry_run=false _model_dir='' _model_dir_mount='' _release_args=() _framework _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'List the images for the specified framework (default='"${DEFAULT_FRAMEWORK}"').\n'
-        exit 0
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run") 
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      -f|--framework)
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1")
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  
-  # Note: Run as user so that the dockerfiles have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) -v $(pwd):/tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  # Call the assembler to list the images
-  for _release_group in "${_release_groups[@]}"
-  do
-    case "$_release_group" in
-      versioned)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-        ;;
-      tf_1.15.2_containers)
-        _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-        ;;
-      *)
-        _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}")
-        ;;
-    esac
-    _model_builder._echo_command ${_asm_dockerfiles} \
-      --release ${_release_group} \
-      --repository ${LOCAL_REPO} \
-      --list_images \
-      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-      ${_release_args[@]} \
-      ${_args[@]} 2>&1 | sort | column -c 2 -t
-    if [[ $_dry_run == false ]]; then
-      if (( ${#_current_releases[@]} > 1 )); then
-        echo ""
-      fi
-    fi
-  done
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Entry point for 'model-builder models'
-#
-_model_builder.models()
-{
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'List all models for the specified framework.\n'
-        exit 0
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo ${_current_models[@]}
-}
-
-#
-# Entry point for 'model-builder packages ...'
-#
-_model_builder.packages()
-{
-  local _abs_model_dir _args=() _asm_dockerfiles _dry_run=false _model_dir='' _model_dir_mount='' _verbose=''
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'List the packages that would be built for the specified framework.\n'
-        exit 0
-        ;;
-      --dry-run)
-        _dry_run=true
-        _args+=("--dry_run")
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      -f|--framework)
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1")
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  
-  # Check for the imz-tf-tools container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  
-  # Note: Run as user so that the dockerfiles have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) $(_model_builder._get-proxy-env-vars) -v $(pwd):/tf $_model_dir_mount imz-tf-tools python3 assembler.py "
-
-  if [[ ${#_current_framework[@]} == 0 ]]; then
-    _current_framework=${DEFAULT_FRAMEWORK}
-  fi
-
-  # Call the assembler to list packages
-  _model_builder._echo_command ${_asm_dockerfiles} \
-    --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-    --repository ${LOCAL_REPO} \
-    --list_packages \
-    --arg _TAG_PREFIX=${TAG_PREFIX} \
-    --arg TENSORFLOW_TAG=${TENSORFLOW_TAG} \
-    ${_args[@]} 2>&1 | sort | column -c 2 -t
-
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Entry point for 'model-builder frameworks'
-#
-_model_builder.frameworks()
-{
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'List all frameworks within the specifications\n'
-        exit 0
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo ${_all_frameworks[@]}
-}
-
-#
-# Called at beginning of test
-#
-setup()
-{
-  # TBD sanity check that command, framework, models are non-zero
-  if (( 0 != 0 )); then
-    echo '# setup' >&3
-  fi
-}
-
-#
-# Called at end of test
-#
-teardown()
-{
-  # TBD sanity check that command, framework, models are non-zero
-  if (( 0 != 0 )); then
-    echo '# teardown' >&3
-  fi
-}
-
-#
-# test function, returns last modified for image
-#
-last_built()
-{
-  local _image _created _os=$(uname -s)
-  _image=$(docker images | grep $1 | head -1 | awk '{printf("%s:%s\n", $1, $2)}')
-  _created="$(docker inspect $_image --format '{{.Created}}' | sed 's/T/ /g' | sed 's/\.[0-9]*Z$//')"
-  case "$_os" in
-    Darwin)
-      echo $(( $(date +%s) - $(date -juf'%Y-%m-%d %H:%M:%S' "$_created" +%s) ))
-      ;;
-    Linux)
-      echo $(( $(date +%s) - $(date --utc --date="$_created" '+%s') ))
-      ;;
-  esac
-}
-
-#
-# test function, returns last modified for file
-#
-last_modified()
-{
-  local _os=$(uname -s)
-  case "$_os" in
-    Darwin)
-      echo $(( $(date +%s) - $(stat -f%c $1) ))
-      ;;
-    Linux)
-      expr $(date +%s) - $(stat -c %Y $1)
-      ;;
-  esac
-}
-
-#
-# Generates tests for model-builder build -f <framework> models...
-#
-_model_builder._tests.build()
-{
-  local _image_name _image_names _model _framework=''
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --arg)
-        shift
-        ;;
-      -h|--help)
-        echo -e 'generate-documentation tests\n'
-        exit 0
-        ;;
-      --model_dir=|--output_dir=|--dry_run)
-        ;;
-      --framework)
-        shift
-        _framework=$1
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo '# *** tests for build ***' >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  _image_names=$(mktemp /tmp/image_names.XXXXXXXXXX)
-  _tmp_files+=($_image_names)
-  model-builder images -f $_framework > $_image_names
-  for _model in "$@"; do
-    _image_name=$(cat $_image_names | grep '^.*'$_model'$' | awk '{print $3}' || exit 0)
-    if [[ -n $_image_name ]]; then
-      echo $'@test "validate build image for '$_model$' in framework '$_framework$' creates '$_image_name$'" {
-  run model-builder build -f '$_framework$' '$_model$'
-  (( $status == 0 ))
-  [[ "${lines[@]}" =~ '$_model$' ]]
-}' >> $_g_root'/tests/model-builder.bats'
-    fi
-  done
-}
-
-#
-# parses model name into parts: _prefix, _precision, _mode
-# also sets the top-level directory to either quickstart or k8s
-#
-_model_builder._parse_model_name()
-{
-  local _model=$1 _framework=$2 _last_suffix _lmode _lprefix _suffix
-  declare -n __prefix=$3
-  declare -n __precision=$4
-  declare -n __mode=$5
-  declare -n __dir=$6
-  if [[ $_model =~ bfloat16 ]]; then
-    __precision=bfloat16
-  elif [[ $_model =~ fp32 ]]; then
-    __precision=fp32
-  elif [[ $_model =~ int8 ]]; then
-    __precision=int8
-  else 
-    echo 'Unknown precision specified in model: '$_model >&2
-    return 1
-  fi
-  _lprefix=${_model%-${__precision}-*}
-  _suffix=${_model/${_lprefix}-/}
-  _lmode=${_suffix#*-}
-  _last_suffix=${_lmode##*-}
-  if [[ $_last_suffix =~ k8s ]]; then
-    __prefix=/${_lprefix//-/[_-]}/
-    __dir=k8s
-  else
-    __prefix=/${_framework}/${_lprefix//-/[_-]}/
-    __dir=quickstart
-  fi
-  __mode=${_lmode%%-*}
-
-  return 0
-}
-
-#
-# Generates tests for model-builder generate-documentation -f <framework> models...
-#
-_model_builder._tests.generate_documentation()
-{
-  local _dir='' _docdir _framework='' _image_name _image_names _mode _model _precision _prefix _re="[[:space:]]+"
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --arg)
-        shift
-        ;;
-      -h|--help)
-        echo -e 'generate-documentation tests\n'
-        exit 0
-        ;;
-      --model_dir=|--output_dir=|--dry_run)
-        ;;
-      --framework)
-        shift
-        _framework=$1
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo '# *** tests for generate-documentation ***' >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  for _model in "$@"; do
-    _model_builder._parse_model_name $_model $_framework _prefix _precision _mode _dir
-    ret_val=$?
-    if (( $ret_val != 0 )); then
-      echo "Skipping validation of generate-documentation for $_model"
-      continue
-    fi
-    _docdir=$(find $_dir -type d | grep $_prefix | grep $_mode | grep $_precision'$')
-    if [[ -n $_docdir ]]; then
-      if [[ $_docdir =~ $_re ]]; then
-        echo "More than one path returned from \`find $_dir -type d ' | grep '$_prefix' | grep '$_mode' | grep '$_precision'$'\`: $(echo $_docdir). Skipping." >&2
-        continue
-      fi
-      _readme=$_docdir/README.md
-      echo $'@test "validate generate-documentation for '$_model$' in framework '$_framework$' creates '$_readme$'" {
-  run model-builder generate-documentation -f '$_framework$' '$_model$'
-  (( $status == 0 ))
-  [[ -f '$_readme$' ]]
-  (( $(last_modified '$_readme$') <= 50 ))
-}' >> $_g_root'/tests/model-builder.bats'
-    fi
-  done
-}
-
-#
-# Generates tests for model-builder generate-dockerfile -f <framework> models...
-#
-_model_builder._tests.generate_dockerfile()
-{
-  local _dockerfile_path _dockerfile _path_length _model _framework _result _results _subdir
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --arg)
-        shift
-        ;;
-      -h|--help)
-        echo -e 'generate-dockerfile tests\n'
-        exit 0
-        ;;
-      --model_dir=|--output_dir=|--dry_run)
-        ;;
-      --framework)
-        shift
-        _framework=$1
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  case "$_framework" in
-    tensorflow)
-      _subdir='{model_containers,dataset_containers}'
-      ;;
-    *)
-      _subdir=$_framework
-      ;;
-  esac
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo '# *** tests for generate-dockerfile ***' >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  for _model in "$@"; do
-    _results=( $(find $(eval "echo dockerfiles/$_subdir") -name '*'$_model'.Dockerfile' 2>/dev/null) )
-    case ${#_results[@]} in 
-      0)
-        _dockerfile_path=''
-        ;;
-      1)
-       _dockerfile_path=${_results[0]}
-       ;;
-      *)
-       # disambiguate between intel-tf-image-segmentation-3d-unet-fp32-inference.Dockerfile and intel-tf-image-segmentation-unet-fp32-inference.Dockerfile
-       _path_length=${#_results[0]}
-       _dockerfile_path=${_results[0]}
-       for _result in "${_results[@]}"; do
-         if (( ${#_result} < $_path_length )); then
-           _dockerfile_path=$_result
-         fi
-       done
-       ;;
-    esac
-    if [[ -n $_dockerfile_path ]]; then
-      _dockerfile=${_dockerfile_path##*/}
-      echo $'@test "validate generate-dockerfile for '$_model$' in framework '$_framework$' creates '$_dockerfile$'" {
-  run model-builder generate-dockerfile -f '$_framework$' '$_model$'
-  (( $status == 0 ))
-  [[ "${lines[@]}" =~ '$_dockerfile$' ]]
-  (( $(last_modified '$_dockerfile_path$') <= 50 ))
-}' >> $_g_root'/tests/model-builder.bats'
-   fi
- done
-}
-
-#
-# Generates tests for model-builder generate-deployment models...
-#
-_model_builder._tests.generate_deployment()
-{
-  local _asm_dockerfiles _model _framework
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --arg)
-        shift
-        ;;
-      -h|--help)
-        echo -e 'generate-deployment tests\n'
-        exit 0
-        ;;
-      --model_dir=|--output_dir=|--dry_run)
-        ;;
-      --framework)
-        shift
-        _framework=$1
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  pushd $_g_root/docker 1>/dev/null 2>/dev/null
-  # Check for the tf-tool container and build it if it doesn't exist
-  _model_builder._check-for-imz-tf-tools $_quiet || _model_builder._build-imz-tf-tools $_g_root/docker
-  _model_dir=$_g_root/..
-  _model_dir_mount='-v $_model_dir:$_model_dir'
-  # Note: Run as user so that the generated deployment(s) have the right permissions
-  _asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) -e MODEL_BUILDER_SPEC_HOME=/specs $(_model_builder._get-proxy-env-vars) -v ${_spec_home}:/specs -v $(pwd):$_model_dir/.. -w $_model_dir/.. $_model_dir_mount imz-tf-tools python3 assembler.py "
-  _model_builder._echo_command ${_asm_dockerfiles} \
-    --release $_release_group \
-    --spec_dir=./specs/$_framework \
-    --generate_deployments_tests \
-    ${_release_args[@]} \
-    ${_args[@]}
-  popd 1>/dev/null 2>/dev/null
-}
-
-#
-# Generates tests for model-builder package -f <framework> models...
-#
-_model_builder._tests.package()
-{
-  local _model _package_names _package_path _package _framework
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --arg)
-        shift
-        ;;
-      -h|--help)
-        echo -e 'package tests\n'
-        exit 0
-        ;;
-      --model_dir=|--output_dir=|--dry_run)
-        ;;
-      --framework)
-        shift
-        _framework=$1
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo '# *** tests for package ***' >> $_g_root'/tests/model-builder.bats'
-  echo '#' >> $_g_root'/tests/model-builder.bats'
-  echo ''  >> $_g_root'/tests/model-builder.bats'
-  _package_names=$(mktemp /tmp/package_names.XXXXXXXXXX)
-  _tmp_files+=($_package_names)
-  model-builder packages -f $_framework 2>&1 > $_package_names
-  for _model in "$@"; do
-    _package_path=$(cat $_package_names | grep '^.*'$_model'.tar.gz$' | awk '{print $2}' || exit 0)
-    if [[ -n $_package_path ]]; then
-      _package=${_package_path##*/}
-      echo $'@test "validate package for '$_model$' in framework '$_framework$' creates '$_package$'" {
-  run model-builder package -f '$_framework$' '$_model$'
-  (( $status == 0 ))
-  [[ "${lines[@]}" =~ '$_package$' ]]
-  (( $(last_modified '$_package_path$') <= 50 ))
-}' >> $_g_root'/tests/model-builder.bats'
-    fi
-  done
-}
-
-_model_builder.run_test_suite()
-{
-  local _abs_model_dir='' _args=() _dry_run='' _model_dir='' _model_dir_mount='' _model_package_folder 
-  local _output_dir='' _release_args=() _framework _frameworks=() _temp_model_package_dir _verbose='' 
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Builds and runs tests for the specified models (default=*) in the specified framework (default='"${_current_framework}"').\n'
-        exit 0
-        ;;
-      -c|--command)
-        ;;
-      --dry-run)
-        _dry_run='--dry_run'
-        ;;
-      --model_dir=*)
-        _abs_model_dir=${1#*=}
-        _model_dir_mount='-v '$_abs_model_dir':/tf/models'
-        _model_dir='--model_dir=models'
-        _args+=("$_model_dir") 
-        ;;
-      --output_dir=*)
-        _output_dir=${1#*=}
-        _output_dir=${_output_dir##*/}
-        _args+=("$1") 
-        ;;
-      -f|--framework)
-        ;;
-      --verbose)
-        _verbose=$1
-        _args+=("$1") 
-        ;;
-      *)
-        _args+=("$1") 
-        ;;
-    esac
-    shift
-  done
-  rm -f $_g_root'/tests/model-builder.bats'
-  type setup | sed '1d' >> $_g_root'/tests/model-builder.bats'
-  type teardown | sed '1d' >> $_g_root'/tests/model-builder.bats'
-  type last_modified | sed '1d' >> $_g_root'/tests/model-builder.bats'
-  echo 'pushd '$(dirname $_g_root) >> $_g_root'/tests/model-builder.bats'
-
-  for _command in "${_current_commands[@]}"
-  do
-    for _release_group in "${_release_groups[@]}"
-    do
-      case "$_release_group" in
-        versioned)
-          _release_args=("--arg" "_TAG_PREFIX=${TAG_PREFIX}" "--arg" "TENSORFLOW_TAG=${TENSORFLOW_TAG}" "--arg" "TENSORFLOW_IMAGE=${TENSORFLOW_IMAGE}")
-          ;;
-        tf_1.15.2_containers)
-          _release_args=("--arg" "_TAG_PREFIX=1.15.2" "--arg" "TENSORFLOW_TAG=1.15.2")
-          ;;
-        *)
-          _release_args=()
-          ;;
-      esac
-    done
-    case "$_command" in
-      build)
-        _model_builder._echo_command \
-          _model_builder._tests.build \
-          --arg http_proxy=${http_proxy} \
-          --arg https_proxy=${https_proxy} \
-          --arg no_proxy=${no_proxy} \
-          --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-          --framework ${_current_framework} \
-          ${_release_args[@]} \
-          ${_args[@]} \
-          $_dry_run \
-          ${_selected_models[@]}
-        ;;
-      generate-documentation)
-        _model_builder._echo_command \
-          _model_builder._tests.generate_documentation \
-          --arg http_proxy=${http_proxy} \
-          --arg https_proxy=${https_proxy} \
-          --arg no_proxy=${no_proxy} \
-          --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-          --framework ${_current_framework} \
-          ${_release_args[@]} \
-          ${_args[@]} \
-          $_dry_run \
-          ${_selected_models[@]}
-        ;;
-      generate-dockerfile)
-        _model_builder._echo_command \
-          _model_builder._tests.generate_dockerfile \
-          --arg http_proxy=${http_proxy} \
-          --arg https_proxy=${https_proxy} \
-          --arg no_proxy=${no_proxy} \
-	      --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-          --framework ${_current_framework} \
-          ${_release_args[@]} \
-          ${_args[@]} \
-          $_dry_run \
-          ${_selected_models[@]}
-        ;;
-      generate-deployment)
-        rm -f $_g_root'/tests/model-builder.bats'
-        _current_framework=k8s
-        _model_builder._echo_command \
-          _model_builder._tests.generate_deployment \
-          --arg http_proxy=${http_proxy} \
-          --arg https_proxy=${https_proxy} \
-          --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-          --framework ${_current_framework} \
-          ${_release_args[@]} \
-          ${_args[@]} \
-          $_dry_run \
-          ${_selected_models[@]}
-        ;;
-      package)
-        _model_builder._echo_command \
-          _model_builder._tests.package \
-          --arg http_proxy=${http_proxy} \
-          --arg https_proxy=${https_proxy} \
-          --arg no_proxy=${no_proxy} \
-          --spec_dir=${_spec_dir_mounted}/${_current_framework} \
-          --framework ${_current_framework} \
-          ${_release_args[@]} \
-          ${_args[@]} \
-          $_dry_run \
-          ${_selected_models[@]}
-        ;;
-    esac
-  done
-
-  _model_builder._echo_command \
-    $_g_root/tests/bin/bats \
-    $_g_root/tests/model-builder.bats \
-    $_dry_run
-
-  result=$?
-  if (( $result != 0 )); then
-    echo "Warning: A non-zero error code ($result) was returned when building the '$_framework' tests"
-  fi
-}
-
-#
-# Calls model-builder completion and redirects its output to /usr/local/etc/bash_completion.d/'$(basename $0)
-#
-_model_builder.install_completion()
-{
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        echo -e 'Call '$(basename $0)' completion > /usr/local/etc/bash_completion.d/'$(basename $0)
-        exit 0
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  if [[ -d /usr/local/etc/bash_completion.d ]]; then
-    $0 completion > /usr/local/etc/bash_completion.d/$(basename $0)
-  fi
-}
-
-#
-# Convenience function: Calls generate-documentation, package, generate-dockerfile, build
-#
-_model_builder.make()
-{
-  # generate documentation
-  _model_builder.generate_documentation $@
-  if (( $? != 0 )); then
-    echo "There was an error while generating documentation." >&2
-    exit 1
-  fi
-
-  # build model packages
-  _model_builder.package $@
-  if (( $? != 0 )); then
-    echo "There was an error building model packages." >&2
-    exit 1
-  fi
-
-  # construct dockerfiles
-  _model_builder.generate_dockerfile $@
-  if (( $? != 0 )); then
-    echo "There was an error while constructing dockerfiles." >&2
-    exit 1
-  fi
-
-  # build images
-  MODEL_PACKAGE_DIR=${MODEL_DIR}/output _model_builder.build $@
-  if (( $? != 0 )); then
-    echo "There was an error while building docker images." >&2
-    exit 1
-  fi
-}
-
-#
-# Parses model args and verifies that the model is 
-# is in the set of _current_models[@]
-#
-_model_builder._validate_models()
-{
-  local _arg
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        return 0
-        ;;
-      -c|--command)
-        shift
-        ;;
-      -f|--framework)
-        shift
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-
-  for _arg in "$@"; do
-    if ! [[ "${_current_models[@]}" =~ $_arg ]]; then
-      echo 'unknown model '$_arg >&2
-      _usage
-      exit 1
-    elif ! [[ "${_selected_models[@]}" =~ $_arg ]]; then
-      _selected_models+=( $_arg )
-      _remaining_models=( ${_remaining_models[@]/$_arg/} )
-    fi
-  done
-  # all models
-  if [[ ${#_selected_models[@]} == 0 ]]; then
-    _selected_models=( ${_current_models[@]}  )
-    _remaining_models=()
-  fi
-}
-
-#
-# Parses args that are frameworks and adds all models within 
-# that framework to _current_models[]
-#
-_model_builder._validate_frameworks()
-{
-  local _arg _args=() _i _index
-  while (( $# > 0 )) && [[ $1 =~ ^- ]]; do
-    case "$1" in
-      -c|--command)
-        shift
-        ;;
-      -h|--help)
-        return 0
-        ;;
-      -f|--framework)
-        _first_framework=true
-        shift
-        _args+=($1)
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  for _index in "${!_args[@]}"; do
-    _arg=${_args[_index]}
-    if [[ $_first_framework == true ]]; then
-        _current_framework=''
-        _current_models=()
-        _remaining_models=()
-        _first_framework=false
-    fi
-    if ! [[ "${_all_frameworks[@]}" =~ $_arg ]]; then
-      echo '_model_builder._validate_frameworks unknown framework '$_arg >&2
-      _usage
-      exit 1
-    fi
-    if [[ "${_all_frameworks[@]}" =~ $_arg ]] && ! [[ "${_current_framework[@]}" =~ $_arg ]]; then
-      _current_framework=$_arg
-      _remaining_frameworks=( ${_remaining_frameworks[@]/$_arg} )
-      _current_models+=( ${_all_models["$_current_framework"]} )
-      _remaining_models+=(${_current_models[@]})
-    fi
-  done
-
-  _release_groups=("versioned")
-  if [[ ${_current_framework} == "tensorflow" ]]; then
-    if [[ ${TAG_PREFIX} != *"centos"* ]] && [[ ${TAG_PREFIX} != *"spr"* ]]; then
-      _release_groups+=("tf_1.15.2_containers")
-    fi
-  fi
-}
-
-#
-# Parses args that are commands and verifies 
-# the command is in _all_commands[] and adds the command 
-# to _current_commands[]
-#
-_model_builder._validate_commands()
-{
-  local _arg _args=($@) _commands=() _first_command=true _i _index
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -h|--help)
-        return 0
-        ;;
-      -c|--command)
-        shift
-        _commands+=($1)
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  for _index in "${!_commands[@]}"; do
-    _arg=${_commands[_index]}
-    if [[ $_first_command == true ]]; then
-      _current_commands=()
-      _first_command=false
-    fi
-    if [[ $_arg == ${_args[@]: -1:1} ]] && (( $_index == ${#_args[@]} - 1 )); then
-      echo ${_remaining_commands[@]}
-      return 1
-    fi
-    if ! [[ "${_all_commands[@]}" =~ $_arg ]]; then
-      echo 'unknown command '$_arg >&2
-      _usage
-      exit 1
-    fi
-    if [[ "${_all_commands[@]}" =~ $_arg ]] && ! [[ "${_current_commands[@]}" =~ $_arg ]]; then
-      _current_commands+=( $_arg )
-      _remaining_commands=( ${_remaining_commands[@]/$_arg/} )
-      _current_framework=${DEFAULT_FRAMEWORK}
-      _model_builder._sort_unique _current_framework
-    fi
-  done
-  if [[ ${#_current_commands[@]} == 0 ]]; then
-    _current_commands=(${_all_commands[@]})
-    _remaining_commands=()
-  fi
-}
-
-#
-# Filters input models as only those in _current_models[@] and returns one or more
-#  ' --only_tags_matching=^.*<model>$ --exclude_tags_matching=^.*<model>$'
-#
-_model_builder._build_only_tags_matching_args()
-{
-  local _arg _curent_models_count _exclude_model _excluded_models=() _exclude_models=() _key _models=()
-  declare -A _models_map
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      -c|--command)
-        shift
-        ;;
-      -h|--help)
-        return 0
-        ;;
-      -f|--framework)
-        shift
-        ;;
-      *)
-        ;;
-    esac
-    shift
-  done
-  _current_models_count=${#_current_models[@]}
-  for _arg in "$@";do
-    if [[ ${_current_models[@]} =~ $_arg ]]; then
-      _models+=(' --only_tags_matching=^.*'$_arg'$')
-      _key=X${_arg//-/_}
-      _models_map[$_key]="true"
-      _exclude_models=(${_current_models[@]//*$_arg} $_arg)
-      if (( $_current_models_count != ${#_exclude_models[@]} )); then
-        _exclude_models=(${_current_models[@]} ${_exclude_models[@]})
-        if (( ${#_exclude_models[@]} > 0 )); then
-          _excluded_models=($(printf "%s\n" "${_exclude_models[@]}" | sort | uniq -u))
-        fi
-      fi
-    fi
-  done
-  _exclude_models=()
-  for _exclude_model in "${_excluded_models[@]}"; do
-    _key=X${_exclude_model//-/_}
-    if [[ -z ${_models_map[$_key]} ]]; then
-      _exclude_models+=(' --exclude_tags_matching=^.*'$_exclude_model'$')
-    fi
-  done
-  echo ${_exclude_models[@]} ${_models[@]} 
-}
-
-#
-# Serializes an associative array (map) so can be reified as a global in a different context
-# example
-#   dict=[["a"]="1 2" ["b"]="3 4" ["c"]="5 6"]
-#   exported as 
-#     'declare -g -A dict=[["a"]="1 2" ["b"]="3 4" ["c"]="5 6"]
-#
-_model_builder._export() 
-{
-  local _name=$1 _exportable=''
-  declare -n _assoc=$2
-  _exportable+='  declare -g -A '$_name'=('
-  for _key in "${!_assoc[@]}"; do
-    _exportable+=$'["'$_key$'"]="'${_assoc[$_key]}$'" '
-  done
-  _exportable+=')'
-  echo "  $_exportable"
-}
-
-#
-# Main entry point for model-builder
-#  handles all options and subcommands
-#
-_model_builder_main()
-{
-  local _dry_run='' _model_dir='--model_dir='$MODEL_DIR _only_tags_matching='' _output_dir='--output_dir=models/output' _subcommand _verbose=''
-
-  while [[ "$#" -gt "0" && $1 =~ ^- ]]; do
-    case "$1" in
-      --dry-run)
-          _dry_run=$1
-          shift
-          ;;
-      -h|--help)
-          _usage
-          exit 0
-          ;;
-      --verbose)
-          _verbose=$1
-          shift
-          ;;
-      *)
-          echo '_model_builder_main unknown option '$1
-          shift
-          exit 1
-          ;;
-    esac
-  done
-
-  case 1 in
-    $(($#==0))) 
-      _usage
-      exit 0
-      ;;
-    $(($#>=1)))
-      _subcommand=$1
-      shift
-      case "$_subcommand" in
-        build)
-          _current_framework=${DEFAULT_FRAMEWORK}
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          MODEL_PACKAGE_DIR=${MODEL_DIR}/output _model_builder.build $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $_nocache $@
-          if (( $? != 0 )); then
-            echo "There was an error while building docker images." >&2
-            exit 1
-          fi
-          ;;
-        commands)
-          _model_builder.commands $@
-          ;;
-        completion)
-          _model_builder.completion $@
-          ;;
-        generate-deployment)
-          _current_framework=k8s
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.generate_deployment $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $@
-          if (( $? != 0 )); then
-            echo "There was an error while generating deployments." >&2
-            exit 1
-          fi
-          ;;
-        generate-dockerfile)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.generate_dockerfile $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $@
-          if (( $? != 0 )); then
-            echo "There was an error while constructing dockerfiles." >&2
-            exit 1
-          fi
-          ;;
-        generate-documentation)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.generate_documentation $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $@
-          if (( $? != 0 )); then
-            echo "There was an error while creating documentation." >&2
-            exit 1
-          fi
-          ;;
-        images)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder.images $_model_dir $_verbose $_dry_run $@
-          ;;
-        init-spec)
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder.init_spec $_verbose $_dry_run $@
-          ;;
-        install-completion)
-          _model_builder.install_completion $@
-          ;;
-        make)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.make $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $@
-          ;;
-        models)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder.models $@
-          ;;
-        package)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.package $_model_dir $_output_dir $_only_tags_matching $_verbose $_dry_run $@
-          if (( $? != 0 )); then
-            echo "There was an error building model packages."
-            exit 1
-          fi
-          ;;
-        packages)
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder.packages $_model_dir $_verbose $_dry_run $@
-          return 0
-          ;;
-        frameworks)
-          _model_builder.frameworks $@
-          ;;
-        run-test-suite)
-          _model_builder._validate_commands $@ || exit 0
-          _model_builder._set_current_models
-          _model_builder._validate_frameworks $@ || exit 0
-          _model_builder._validate_models $@ || exit 0
-          _only_tags_matching="$(_model_builder._build_only_tags_matching_args $@)"
-          _model_builder.run_test_suite $_only_tags_matching $_verbose $_dry_run $@
-          if (( $? != 0 )); then
-            echo "There was an error while running tests."
-            exit 1
-          fi
-          ;;
-        *)
-          echo 'unknown option '$1' '$@ >&2
-          exit 1
-          ;;
-      esac
-      ;;
-  esac
-}
-
-_model_builder._init $@
-_model_builder_main $@
diff --git a/tools/tests/README.md b/tools/tests/README.md
deleted file mode 100644
index 90894cb5e..000000000
--- a/tools/tests/README.md
+++ /dev/null
@@ -1,93 +0,0 @@
-
-# Tests for models 
-
-## Moving towards a [GitOps](https://about.gitlab.com/topics/gitops/) paradigm
-
-1. Model testing will use the git repository as the source of truth
-1. Model tests will be declarative, based on model specifications versioned in git under docker/specs
-1. Model tests will be generated as test artifacts from these model specifications
-1. Model tests and test outputs will be compliant with the [Test Anything Protocol (TAP)](http://testanything.org/).
-
-
-## CICD
-
-As part of Jenkins CICD the following steps will be included for MRs and general regression testing:
-
-1. Call `model-builder run-test-suite`
-
-   This subcommand will call model-builder-test-suite-generator which will auto-discover all models by calling `model-builder models`.
-   This call returns all models declared under `tools/docker/specs`. Each model's manifest includes information
-   used to generate a Dockerfile, build a container and package the model. The generator will enumerate over the set of models
-   and generate a set of tests for each model. These tests validate model-builder subcommands and are appended to model-builder.bats.
-   Currently, the automated tests cover model-builder subcommands: generate-dockerfile, generate-documentation, generate-deployment, build, and package.
-   Additional tests related to model creation (model-builder init-spec) and model deployment (docker, k8s) will be added over time.
-
-   These tests look like:
-
-```
-@test "validate generate-dockerfile for bert-large-fp32-training creates intel-tf-language-modeling-bert-large-fp32-training.Dockerfile" {
-    run model-builder -q generate-dockerfile bert-large-fp32-training
-    (( $status == 0 ))
-    [[ ${lines[@]} =~ intel-tf-language-modeling-bert-large-fp32-training.Dockerfile ]]
-    (( $(last_modified dockerfiles/model_containers/intel-tf-language-modeling-bert-large-fp32-training.Dockerfile) <= 50 ))
-  }
-@test "validate generate-dockerfile for image-recognition creates intel-tf-image-recognition.Dockerfile" {
-    run model-builder -q generate-dockerfile image-recognition
-    (( $status == 0 ))
-    [[ ${lines[@]} =~ intel-tf-image-recognition.Dockerfile ]]
-    (( $(last_modified dockerfiles/intel-tf-image-recognition.Dockerfile) <= 50 ))
-  }
-```
-
-  The test output (compliant with [Test Anything Protocol (TAP)](http://testanything.org/)) looks like:
-
-```
- ✓ validate 'model-builder commands'
- ✓ validate 'model-builder models'
- ✓ validate generate-dockerfile for bert-large-fp32-training creates intel-tf-language-modeling-bert-large-fp32-training.Dockerfile
- ✓ validate generate-dockerfile for image-recognition creates intel-tf-image-recognition.Dockerfile
- ✓ validate generate-dockerfile for language-modeling creates intel-tf-language-modeling.Dockerfile
- ✓ validate generate-dockerfile for object-detection creates intel-tf-object-detection.Dockerfile
- ✓ validate generate-dockerfile for preprocess-coco-val creates intel-tf-object-detection-preprocess-coco-val.Dockerfile
-```
-
-### K8s Model deployment tests:
-
-The command `model-builder run-test-suite -c generate-deployment -f k8s <target-k8s-model>`
-generates and tests kubernetes deployments for the target model using the model's k8 package such as `bert-large-fp32-training-k8s`. The supported K8s models are listed in `models/k8s`.
-
->Note: 
-> * If $HOME is a nfs mount point then you should chose an alternate directory eg: `XDG_CONFIG_HOME=/tmp/$USER/config model-builder run-test-suite generate-deployment ...`.
-> * The user can edit the [k8 specification files](models/tools/docker/specs/k8s) to provide env values specific to their k8 deployment. For example the user would replace USER_ID with their [id](https://man7.org/linux/man-pages/man1/id.1.html).
-
-`generate-deployment` in `model-builder-test-suite-generator` aggregates a list of test snippets based on the model specification's `runtime.tests` and append them to `model-builder.bats`
-
-Example for the bert-large-fp32-training-k8s specification's `runtime.tests`:
-```
-        tests:
-          - uri: k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/deployments-yml-exist.sh
-            args:
-              - name: ''
-                value: ''
-          - uri: k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/namespace-exists.sh
-            args:
-              - name: USER_NAME
-                value: $USER_NAME
-          - uri: k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests/single_node_deployment.sh
-            args:
-              - name: ''
-                value: ''
-```
-`uri` refers to each test snippet location in the `.tests` such as `models/k8s/language_modeling/tensorflow/bert_large/training/fp32/.tests`.
-
-The test output looks like:
-```
- ✓ validate package for bert-large-fp32-training-k8s in framework k8s creates model package file 
- ✓ validate generate-deployment for bert-large-fp32-training-k8s in framework k8s creates deployment files under deployments/bert-large-fp32-training-k8s/ 
- ✓ verify namespace <USER_NAME> exists 
- ✓ kubectl apply of single-node bert-large-fp32-training-k8s 
- ✓ single-node deployment of bert-large-fp32-training-k8s creates 1 pod and it is running 
- ✓ kubectl delete of single-node bert-large-fp32-training-k8s 
-
-6 tests, 0 failures
-```
\ No newline at end of file
diff --git a/tools/tests/bin/bats b/tools/tests/bin/bats
deleted file mode 100755
index bc83be0be..000000000
--- a/tools/tests/bin/bats
+++ /dev/null
@@ -1,77 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-set -euo pipefail
-
-if command -v greadlink >/dev/null; then
-  bats_readlinkf() {
-    greadlink -f "$1"
-  }
-else
-  bats_readlinkf() {
-    readlink -f "$1"
-  }
-fi
-
-fallback_to_readlinkf_posix() {
-  bats_readlinkf() {
-    [ "${1:-}" ] || return 1
-    max_symlinks=40
-    CDPATH='' # to avoid changing to an unexpected directory
-
-    target=$1
-    [ -e "${target%/}" ] || target=${1%"${1##*[!/]}"} # trim trailing slashes
-    [ -d "${target:-/}" ] && target="$target/"
-
-    cd -P . 2>/dev/null || return 1
-    while [ "$max_symlinks" -ge 0 ] && max_symlinks=$((max_symlinks - 1)); do
-      if [ ! "$target" = "${target%/*}" ]; then
-        case $target in
-          /*) cd -P "${target%/*}/" 2>/dev/null || break ;;
-          *) cd -P "./${target%/*}" 2>/dev/null || break ;;
-        esac
-        target=${target##*/}
-      fi
-
-      if [ ! -L "$target" ]; then
-        target="${PWD%/}${target:+/}${target}"
-        printf '%s\n' "${target:-/}"
-        return 0
-      fi
-
-      # `ls -dl` format: "%s %u %s %s %u %s %s -> %s\n",
-      #   <file mode>, <number of links>, <owner name>, <group name>,
-      #   <size>, <date and time>, <pathname of link>, <contents of link>
-      # https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html
-      link=$(ls -dl -- "$target" 2>/dev/null) || break
-      target=${link#*" $target -> "}
-    done
-    return 1
-  }
-}
-
-if ! BATS_PATH=$(bats_readlinkf "${BASH_SOURCE[0]}" 2>/dev/null); then
-  fallback_to_readlinkf_posix
-  BATS_PATH=$(bats_readlinkf "${BASH_SOURCE[0]}")
-fi
-
-export BATS_ROOT=${BATS_PATH%/*/*}
-export -f bats_readlinkf
-exec env BATS_ROOT="$BATS_ROOT" "$BATS_ROOT/libexec/bats-core/bats" "$@"
diff --git a/tools/tests/lib/bats-core/preprocessing.bash b/tools/tests/lib/bats-core/preprocessing.bash
deleted file mode 100755
index 282a06730..000000000
--- a/tools/tests/lib/bats-core/preprocessing.bash
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-if [[ -z "$TMPDIR" ]]; then
-	BATS_TMPDIR='/tmp'
-else
-	BATS_TMPDIR="${TMPDIR%/}"
-fi
-
-BATS_TMPNAME="$BATS_RUN_TMPDIR/bats.$$"
-BATS_PARENT_TMPNAME="$BATS_RUN_TMPDIR/bats.$PPID"
-# shellcheck disable=SC2034
-BATS_OUT="${BATS_TMPNAME}.out" # used in bats-exec-file
-
-bats_preprocess_source() {
-	BATS_TEST_SOURCE="${BATS_TMPNAME}.src"
-	bats-preprocess "$BATS_TEST_FILENAME" >"$BATS_TEST_SOURCE"
-	trap 'bats_cleanup_preprocessed_source' ERR EXIT
-	trap 'bats_cleanup_preprocessed_source; exit 1' INT
-}
-
-bats_cleanup_preprocessed_source() {
-	rm -f "$BATS_TEST_SOURCE"
-}
-
-bats_evaluate_preprocessed_source() {
-	if [[ -z "$BATS_TEST_SOURCE" ]]; then
-		BATS_TEST_SOURCE="${BATS_PARENT_TMPNAME}.src"
-	fi
-	# Dynamically loaded user files provided outside of Bats.
-	# shellcheck disable=SC1090
-	source "$BATS_TEST_SOURCE"
-}
diff --git a/tools/tests/lib/bats-core/test_functions.bash b/tools/tests/lib/bats-core/test_functions.bash
deleted file mode 100755
index e9ed63e54..000000000
--- a/tools/tests/lib/bats-core/test_functions.bash
+++ /dev/null
@@ -1,109 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-BATS_TEST_DIRNAME="${BATS_TEST_FILENAME%/*}"
-BATS_TEST_NAMES=()
-
-# Shorthand for source-ing files relative to the BATS_TEST_DIRNAME,
-# optionally with a .bash suffix appended. If the argument doesn't
-# resolve relative to BATS_TEST_DIRNAME it is sourced as-is.
-load() {
-  local file="${1:?}"
-
-  # For backwards-compatibility first look for a .bash-suffixed file.
-  # TODO consider flipping the order here; it would be more consistent
-  # and less surprising to look for an exact-match first.
-  if [[ -f "${BATS_TEST_DIRNAME}/${file}.bash" ]]; then
-    file="${BATS_TEST_DIRNAME}/${file}.bash"
-  elif [[ -f "${BATS_TEST_DIRNAME}/${file}" ]]; then
-    file="${BATS_TEST_DIRNAME}/${file}"
-  fi
-
-  if [[ ! -f "$file" ]] && ! type -P "$file" >/dev/null; then
-    printf 'bats: %s does not exist\n' "$file" >&2
-    exit 1
-  fi
-
-  # Dynamically loaded user file provided outside of Bats.
-  # Note: 'source "$file" || exit' doesn't work on bash3.2.
-  # shellcheck disable=SC1090
-  source "${file}"
-}
-
-run() {
-  local origFlags="$-"
-  set +eET
-  local origIFS="$IFS"
-  # 'output', 'status', 'lines' are global variables available to tests.
-  # shellcheck disable=SC2034
-  output="$("$@" 2>&1)"
-  # shellcheck disable=SC2034
-  status="$?"
-  # shellcheck disable=SC2034,SC2206
-  IFS=$'\n' lines=($output)
-  IFS="$origIFS"
-  set "-$origFlags"
-}
-
-setup() {
-  return 0
-}
-
-teardown() {
-  return 0
-}
-
-skip() {
-  # if this is a skip in teardown ...
-  if [[ -n "$BATS_TEARDOWN_STARTED" ]]; then
-    # ... we want to skip the rest of teardown.
-    # communicate to bats_exit_trap that the teardown was completed without error
-    # shellcheck disable=SC2034
-    BATS_TEARDOWN_COMPLETED=1
-    # if we are already in the exit trap (e.g. due to previous skip) ...
-    if [[ "$BATS_TEARDOWN_STARTED" == as-exit-trap ]]; then
-      # ... we need to do the rest of the tear_down_trap that would otherwise be skipped after the next call to exit
-      bats_exit_trap
-      # and then do the exit (at the end of this function)
-    fi
-    # if we aren't in exit trap, the normal exit handling should suffice
-  else
-    # ... this is either skip in test or skip in setup.
-    # Following variables are used in bats-exec-test which sources this file
-    # shellcheck disable=SC2034
-    BATS_TEST_SKIPPED="${1:-1}"
-    # shellcheck disable=SC2034
-    BATS_TEST_COMPLETED=1
-  fi
-  exit 0
-}
-
-bats_test_begin() {
-  BATS_TEST_DESCRIPTION="$1"
-  if [[ -n "$BATS_EXTENDED_SYNTAX" ]]; then
-    printf 'begin %d %s\n' "$BATS_TEST_NUMBER" "$BATS_TEST_DESCRIPTION" >&3
-  fi
-  setup
-}
-
-bats_test_function() {
-  local test_name="$1"
-  BATS_TEST_NAMES+=("$test_name")
-}
diff --git a/tools/tests/lib/bats-core/tracing.bash b/tools/tests/lib/bats-core/tracing.bash
deleted file mode 100755
index 71f0652c5..000000000
--- a/tools/tests/lib/bats-core/tracing.bash
+++ /dev/null
@@ -1,172 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-bats_capture_stack_trace() {
-	local test_file
-	local funcname
-	local i
-
-	BATS_STACK_TRACE=()
-
-	for ((i = 2; i != ${#FUNCNAME[@]}; ++i)); do
-		# Use BATS_TEST_SOURCE if necessary to work around Bash < 4.4 bug whereby
-		# calling an exported function erases the test file's BASH_SOURCE entry.
-		test_file="${BASH_SOURCE[$i]:-$BATS_TEST_SOURCE}"
-		funcname="${FUNCNAME[$i]}"
-		BATS_STACK_TRACE+=("${BASH_LINENO[$((i - 1))]} $funcname $test_file")
-		if [[ "$test_file" == "$BATS_TEST_SOURCE" ]]; then
-			case "$funcname" in
-			"$BATS_TEST_NAME" | setup | teardown | setup_file | teardown_file)
-				break
-				;;
-			esac
-		fi
-	done
-}
-
-bats_print_stack_trace() {
-	local frame
-	local index=1
-	local count="${#@}"
-	local filename
-	local lineno
-
-	for frame in "$@"; do
-		bats_frame_filename "$frame" 'filename'
-		bats_trim_filename "$filename" 'filename'
-		bats_frame_lineno "$frame" 'lineno'
-
-		if [[ $index -eq 1 ]]; then
-			printf '# ('
-		else
-			printf '#  '
-		fi
-
-		local fn
-		bats_frame_function "$frame" 'fn'
-		if [[ "$fn" != "$BATS_TEST_NAME" ]]; then
-			printf "from function \`%s' " "$fn"
-		fi
-
-		if [[ $index -eq $count ]]; then
-			printf 'in test file %s, line %d)\n' "$filename" "$lineno"
-		else
-			printf 'in file %s, line %d,\n' "$filename" "$lineno"
-		fi
-
-		((++index))
-	done
-}
-
-bats_print_failed_command() {
-	local frame="${BATS_STACK_TRACE[${#BATS_STACK_TRACE[@]} - 1]}"
-	local filename
-	local lineno
-	local failed_line
-	local failed_command
-
-	bats_frame_filename "$frame" 'filename'
-	bats_frame_lineno "$frame" 'lineno'
-	bats_extract_line "$filename" "$lineno" 'failed_line'
-	bats_strip_string "$failed_line" 'failed_command'
-	printf '%s' "#   \`${failed_command}' "
-
-	if [[ "$BATS_ERROR_STATUS" -eq 1 ]]; then
-		printf 'failed\n'
-	else
-		printf 'failed with status %d\n' "$BATS_ERROR_STATUS"
-	fi
-}
-
-bats_frame_lineno() {
-	printf -v "$2" '%s' "${1%% *}"
-}
-
-bats_frame_function() {
-	local __bff_function="${1#* }"
-	printf -v "$2" '%s' "${__bff_function%% *}"
-}
-
-bats_frame_filename() {
-	local __bff_filename="${1#* }"
-	__bff_filename="${__bff_filename#* }"
-
-	if [[ "$__bff_filename" == "$BATS_TEST_SOURCE" ]]; then
-		__bff_filename="$BATS_TEST_FILENAME"
-	fi
-	printf -v "$2" '%s' "$__bff_filename"
-}
-
-bats_extract_line() {
-	local __bats_extract_line_line
-	local __bats_extract_line_index=0
-
-	while IFS= read -r __bats_extract_line_line; do
-		if [[ "$((++__bats_extract_line_index))" -eq "$2" ]]; then
-			printf -v "$3" '%s' "${__bats_extract_line_line%$'\r'}"
-			break
-		fi
-	done <"$1"
-}
-
-bats_strip_string() {
-	[[ "$1" =~ ^[[:space:]]*(.*)[[:space:]]*$ ]]
-	printf -v "$2" '%s' "${BASH_REMATCH[1]}"
-}
-
-bats_trim_filename() {
-	printf -v "$2" '%s' "${1#$BATS_CWD/}"
-}
-
-bats_debug_trap() {
-	# don't update the trace within library functions or we get backtraces from inside traps
-	if [[ "$1" != $BATS_ROOT/lib/* && "$1" != $BATS_ROOT/libexec/* ]]; then
-		# The last entry in the stack trace is not useful when en error occured:
-		# It is either duplicated (kinda correct) or has wrong line number (Bash < 4.4)
-		# Therefore we capture the stacktrace but use it only after the next debug
-		# trap fired.
-		# Expansion is required for empty arrays which otherwise error
-		BATS_CURRENT_STACK_TRACE=("${BATS_STACK_TRACE[@]+"${BATS_STACK_TRACE[@]}"}")
-		bats_capture_stack_trace
-	fi
-}
-
-# For some versions of Bash, the `ERR` trap may not always fire for every
-# command failure, but the `EXIT` trap will. Also, some command failures may not
-# set `$?` properly. See #72 and #81 for details.
-#
-# For this reason, we call `bats_error_trap` at the very beginning of
-# `bats_teardown_trap` (the `DEBUG` trap for the call will fix the stack trace)
-# and check the value of `$BATS_TEST_COMPLETED` before taking other actions.
-# We also adjust the exit status value if needed.
-#
-# See `bats_exit_trap` for an additional EXIT error handling case when `$?`
-# isn't set properly during `teardown()` errors.
-bats_error_trap() {
-	local status="$?"
-	if [[ -z "$BATS_TEST_COMPLETED" ]]; then
-		BATS_ERROR_STATUS="${BATS_ERROR_STATUS:-$status}"
-		if [[ "$BATS_ERROR_STATUS" -eq 0 ]]; then
-			BATS_ERROR_STATUS=1
-		fi
-		BATS_STACK_TRACE=("${BATS_CURRENT_STACK_TRACE[@]}")
-		trap - DEBUG
-	fi
-}
diff --git a/tools/tests/lib/bats-core/validator.bash b/tools/tests/lib/bats-core/validator.bash
deleted file mode 100755
index 8ecd9b305..000000000
--- a/tools/tests/lib/bats-core/validator.bash
+++ /dev/null
@@ -1,55 +0,0 @@
-#!/usr/bin/env bash
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-bats_test_count_validator() {
-  header_pattern='[0-9]+\.\.[0-9]+'
-  IFS= read -r header
-  # repeat the header
-  printf "%s\n" "$header"
-
-  # if we detect a TAP plan
-  if [[ "$header" =~ $header_pattern ]]; then
-    # extract the number of tests ...
-    local expected_number_of_tests="${header:3}"
-    # ... count the actual number of [not ] oks...
-    local actual_number_of_tests=0
-    while IFS= read -r line; do
-        # forward line
-        printf "%s\n" "$line"
-        case "$line" in
-        'ok '*)
-        (( ++actual_number_of_tests ))
-        ;;
-        'not ok'*)
-        (( ++actual_number_of_tests ))
-        ;;
-        esac
-    done
-    # ... and error if they are not the same
-    if [[ "${actual_number_of_tests}" != "${expected_number_of_tests}" ]]; then
-        printf '# bats warning: Executed %s instead of expected %s tests\n' "$actual_number_of_tests" "$expected_number_of_tests"
-        return 1
-    fi
-  else
-    # forward output unchanged
-    cat
-  fi
-}
\ No newline at end of file
diff --git a/tools/tests/lib/detik.bash b/tools/tests/lib/detik.bash
deleted file mode 100755
index 4d39312f0..000000000
--- a/tools/tests/lib/detik.bash
+++ /dev/null
@@ -1,303 +0,0 @@
-#!/bin/bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-directory=$(dirname "${BASH_SOURCE[0]}")
-source "$directory/utils.bash"
-
-
-# Retrieves values and attempts to compare values to an expected result (with retries).
-# @param {string} A text query that respect the appropriate syntax
-# @return
-#	1 Empty query
-#	2 Invalid syntax
-#	3 The assertion could not be verified after all the attempts
-#	  (may also indicate an error with the K8s client)
-#	0 Everything is fine
-try() {
-
-	# Concatenate all the arguments into a single string
-	IFS=' '
-	exp="$*"
-
-	# Trim the expression
-	exp=$(trim "$exp")
-
-	# Make the regular expression case-insensitive
-	shopt -s nocasematch;
-
-	# Verify the expression and use it to build a request
-	if [[ "$exp" == "" ]]; then
-		echo "An empty expression was not expected."
-		return 1
-	fi
-
-	# Let's verify the syntax
-	times=""
-	delay=""
-	resource=""
-	name=""
-	property=""
-	expected_value=""
-	expected_count=""
-
-	if [[ "$exp" =~ $try_regex_verify ]]; then
-
-		# Extract parameters
-		times="${BASH_REMATCH[1]}"
-		delay="${BASH_REMATCH[2]}"
-		resource=$(to_lower_case "${BASH_REMATCH[3]}")
-		name="${BASH_REMATCH[4]}"
-		property="${BASH_REMATCH[5]}"
-		expected_value=$(to_lower_case "${BASH_REMATCH[6]}")
-
-	elif [[ "$exp" =~ $try_regex_find ]]; then
-
-		# Extract parameters
-		times="${BASH_REMATCH[1]}"
-		delay="${BASH_REMATCH[2]}"
-		expected_count="${BASH_REMATCH[3]}"
-		resource=$(to_lower_case "${BASH_REMATCH[4]}")
-		name="${BASH_REMATCH[5]}"
-		property="${BASH_REMATCH[6]}"
-		expected_value=$(to_lower_case "${BASH_REMATCH[7]}")
-	fi
-
-	# Do we have something?
-	if [[ "$times" != "" ]]; then
-
-		# Prevent line breaks from being removed in command results
-		IFS=""
-
-		# Start the loop
-		echo "Valid expression. Verification in progress..."
-		code=0
-		for ((i=1; i<=$times; i++)); do
-
-			# Verify the value
-			verify_value $property $expected_value $resource $name "$expected_count"
-			code=$?
-
-			# Break the loop prematurely?
-			if [[ "$code" == "0" ]]; then
-				break
-			elif [[ "$i" != "1" ]]; then
-				code=3
-				sleep $delay
-			else
-				code=3
-			fi
-		done
-
-		## Error code
-		return $code
-	fi
-
-	# Default behavior
-	echo "Invalid expression: it does not respect the expected syntax."
-	return 2
-}
-
-
-# Retrieves values and attempts to compare values to an expected result (without any retry).
-# @param {string} A text query that respect one of the supported syntaxes
-# @return
-#	1 Empty query
-#	2 Invalid syntax
-#	3 The elements count is incorrect
-#	  (may also indicate an error with the K8s client)
-#	0 Everything is fine
-verify() {
-
-	# Concatenate all the arguments into a single string
-	IFS=' '
-	exp="$*"
-
-	# Trim the expression
-	exp=$(trim "$exp")
-
-	# Make the regular expression case-insensitive
-	shopt -s nocasematch;
-
-	# Verify the expression and use it to build a request
-	if [[ "$exp" == "" ]]; then
-		echo "An empty expression was not expected."
-		return 1
-
-	elif [[ "$exp" =~ $verify_regex_count_is ]] || [[ "$exp" =~ $verify_regex_count_are ]]; then
-		card="${BASH_REMATCH[1]}"
-		resource=$(to_lower_case "${BASH_REMATCH[2]}")
-		name="${BASH_REMATCH[3]}"
-
-		echo "Valid expression. Verification in progress..."
-		query=$(build_k8s_request "")
-		client_with_options=$(build_k8s_client_with_options)
-		result=$(eval $client_with_options get $resource $query | grep $name | tail -n +1 | wc -l | tr -d '[:space:]')
-
-		# Debug?
-		detik_debug "-----DETIK:begin-----"
-		detik_debug "$BATS_TEST_FILENAME"
-		detik_debug "$BATS_TEST_DESCRIPTION"
-		detik_debug ""
-		detik_debug "Client query:"
-		detik_debug "$client_with_options get $resource $query"
-		detik_debug ""
-		detik_debug "Result:"
-		detik_debug "$result"
-		detik_debug "-----DETIK:end-----"
-		detik_debug ""
-
-		if [[ "$result" == "$card" ]]; then
-			echo "Found $result $resource named $name (as expected)."
-		else
-			echo "Found $result $resource named $name (instead of $card expected)."
-			return 3
-		fi
-
-	elif [[ "$exp" =~ $verify_regex_property_is ]]; then
-		property="${BASH_REMATCH[1]}"
-		expected_value="${BASH_REMATCH[2]}"
-		resource=$(to_lower_case "${BASH_REMATCH[3]}")
-		name="${BASH_REMATCH[4]}"
-
-		echo "Valid expression. Verification in progress..."
-		verify_value $property $expected_value $resource $name
-
-		if [[ "$?" != "0" ]]; then
-			return 3
-		fi
-
-	else
-		echo "Invalid expression: it does not respect the expected syntax."
-		return 2
-	fi
-}
-
-
-# Verifies the value of a column for a set of elements.
-# @param {string} A K8s column or one of the supported aliases.
-# @param {string} The expected value.
-# @param {string} The resouce type (e.g. pod).
-# @param {string} The resource name or regex.
-# @param {integer} a.k.a. "expected_count": the expected number of elements having this property (optional)
-# @return
-# 		If "expected_count" was NOT set: the number of elements with the wrong value.
-#		If "expected_count" was set: 101 if the elements count is not right, 0 otherwise.
-verify_value() {
-
-	# Make the parameters readable
-	property="$1"
-	expected_value=$(to_lower_case "$2")
-	resource="$3"
-	name="$4"
-	expected_count="$5"
-
-	# List the items and remove the first line (the one that contains the column names)
-	query=$(build_k8s_request $property)
-	client_with_options=$(build_k8s_client_with_options)
-	result=$(eval $client_with_options get $resource $query | grep $name | tail -n +1)
-
-	# Debug?
-	detik_debug "-----DETIK:begin-----"
-	detik_debug "$BATS_TEST_FILENAME"
-	detik_debug "$BATS_TEST_DESCRIPTION"
-	detik_debug ""
-	detik_debug "Client query:"
-	detik_debug "$client_with_options get $resource $query"
-	detik_debug ""
-	detik_debug "Result:"
-	detik_debug "$result"
-	if [[ "$expected_count" != "" ]]; then
-		detik_debug ""
-		detik_debug "Expected count: $expected_count"
-	fi
-	detik_debug "-----DETIK:end-----"
-	detik_debug ""
-
-	# Is the result empty?
-	empty=0
-	if [[ "$result" == "" ]]; then
-		echo "No resource of type '$resource' was found with the name '$name'."
-	fi
-
-	# Verify the result
-	IFS=$'\n'
-	invalid=0
-	valid=0
-	for line in $result; do
-
-		# Keep the second column (property to verify)
-		# and put it in lower case
-		value=$(to_lower_case "$line" | awk '{ print $2 }')
-		element=$(echo "$line" | awk '{ print $1 }')
-		if [[ "$value" != "$expected_value" ]]; then
-			echo "Current value for $element is $value..."
-			invalid=$((invalid + 1))
-		else
-			echo "$element has the right value ($value)."
-			valid=$((valid + 1))
-		fi
-	done
-
-	# Do we have the right number of elements?
-	if [[ "$expected_count" != "" ]]; then
-		if [[ "$valid" != "$expected_count" ]]; then
-			echo "Expected $expected_count $resource named $name to have this value ($expected_value). Found $valid."
-			invalid=101
-		else
-			invalid=0
-		fi
-	fi
-
-	return $invalid
-}
-
-
-# Builds the request for the get operation of the K8s client.
-# @param {string} A K8s column or one of the supported aliases.
-# @return 0
-build_k8s_request() {
-
-	req="-o custom-columns=NAME:.metadata.name"
-	if [[ "$1" == "status" ]]; then
-		req="$req,PROP:.status.phase"
-	elif [[ "$1" == "port" ]]; then
-		req="$req,PROP:.spec.ports[*].port"
-	elif [[ "$1" == "targetPort" ]]; then
-		req="$req,PROP:.spec.ports[*].targetPort"
-	elif [[ "$1" != "" ]]; then
-		req="$req,PROP:$1"
-	fi
-
-	echo $req
-}
-
-
-# Builds the client command, with the option for the K8s namespace, if any.
-# @return 0
-build_k8s_client_with_options() {
-
-	client_with_options="$DETIK_CLIENT_NAME"
-	if [[ ! -z "$DETIK_CLIENT_NAMESPACE" ]]; then
-		# eval does not like '-n'
-		client_with_options="$DETIK_CLIENT_NAME --namespace=$DETIK_CLIENT_NAMESPACE"
-	fi
-
-	echo $client_with_options
-}
diff --git a/tools/tests/lib/test_helper.bash b/tools/tests/lib/test_helper.bash
deleted file mode 100644
index 2507b9e07..000000000
--- a/tools/tests/lib/test_helper.bash
+++ /dev/null
@@ -1,59 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-load "lib/utils"
-load "lib/detik"
-
-DETIK_CLIENT_NAME="kubectl"
-
-# create namespace $USER if not exist
-user_name="$USER"
-echo "-------------------------------------------"
-if [[ $user_name =~ _ ]]; then
-  user_name="${user_name//_/-}"
-  echo " "
-  echo "*** Cannot create namespace for USERNAME $USER contains '_'"
-  echo "*** Tests will use USERNAME $user_name instead ..."
-fi
-
-(kubectl get ns -oname | grep $user_name || kubectl create ns $user_name) && $DETIK_CLIENT_NAME config set-context $($DETIK_CLIENT_NAME config current-context) --namespace=$user_name
-echo " "
-
-last_modified ()
-{
-    local _os=$(uname -s);
-    case "$_os" in
-        Darwin)
-            echo $(( $(date +%s) - $(stat -f%c $1) ))
-        ;;
-        Linux)
-            expr $(date +%s) - $(stat -c %Y $1)
-        ;;
-    esac
-}
-
-launcher_logs()
-{
-  local _launcher_pod=$($DETIK_CLIENT_NAME get pods --no-headers -oname | grep launcher | sed 's^pod/^^')
-  if [[ -n $_launcher_pod ]]; then
-    $DETIK_CLIENT_NAME logs $_launcher_pod | tail -5
-  fi
-}
-# *** tests for generate-deployment ***
\ No newline at end of file
diff --git a/tools/tests/lib/utils.bash b/tools/tests/lib/utils.bash
deleted file mode 100755
index dcac92b9b..000000000
--- a/tools/tests/lib/utils.bash
+++ /dev/null
@@ -1,93 +0,0 @@
-#!/bin/bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-
-# The regex for the "try" key word
-try_regex_verify="^at +most +([0-9]+) +times +every +([0-9]+)s +to +get +([a-z]+) +named +'([^']+)' +and +verify +that +'([^']+)' +is +'([^']+)'$"
-try_regex_find="^at +most +([0-9]+) +times +every +([0-9]+)s +to +find +([0-9]+) +([a-z]+) +named +'([^']+)' +with +'([^']+)' +being +'([^']+)'$"
-
-# The regex for the "verify" key word
-verify_regex_count_is="^there +is +(0|1) +([a-z]+) +named +'([^']+)'$"
-verify_regex_count_are="^there +are +([0-9]+) +([a-z]+) +named +'([^']+)'$"
-verify_regex_property_is="^'([^']+)' +is +'([^']+)' +for +([a-z]+) +named +'([^']+)'$"
-
-
-
-# Prints a string in lower case.
-# @param {string} The string.
-# @return 0
-to_lower_case() {
-	echo "$1" | tr '[:upper:]' '[:lower:]'
-}
-
-
-# Trims a text.
-# @param {string} The string.
-# @return 0
-trim() {
-	echo $1 | sed -e 's/^[[:space:]]*([^[[:space:]]].*[^[[:space:]]])[[:space:]]*$/$1/'
-}
-
-
-# Trims ANSI codes (used to format strings in consoles).
-# @param {string} The string.
-# @return 0
-trim_ansi_codes() {
-	echo $1 | sed -e 's/[[:cntrl:]]\[[0-9;]*[a-zA-Z]//g'
-}
-
-
-# Adds a debug message for a given test.
-# @param {string} The debug message.
-# @return 0
-debug() {
-	debug_filename=$(basename -- $BATS_TEST_FILENAME)
-	mkdir -p /tmp/detik
-	echo -e "$1" >> "/tmp/detik/$debug_filename.debug"
-}
-
-
-# Deletes the file that contains debug messages for a given test.
-# @return 0
-reset_debug() {
-	debug_filename=$(basename -- $BATS_TEST_FILENAME)
-	rm -f "/tmp/detik/$debug_filename.debug"
-}
-
-
-# Adds a debug message for a given test about DETIK.
-# @param {string} The debug message.
-# @return 0
-detik_debug() {
-
-	if [[ "$DEBUG_DETIK" == "true" ]]; then
-		debug "$1"
-        fi
-}
-
-
-# Deletes the file that contains debug messages for a given test about DETIK.
-# @return 0
-reset_detik_debug() {
-
-	if [[ "$DEBUG_DETIK" == "true" ]]; then
-		reset_debug
-	fi
-}
diff --git a/tools/tests/libexec/bats-core/bats b/tools/tests/libexec/bats-core/bats
deleted file mode 100755
index 28f0ec541..000000000
--- a/tools/tests/libexec/bats-core/bats
+++ /dev/null
@@ -1,252 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-export BATS_VERSION='1.2.1'
-
-version() {
-  printf 'Bats %s\n' "$BATS_VERSION"
-}
-
-abort() {
-  printf 'Error: %s\n' "$1" >&2
-  usage >&2
-  exit 1
-}
-
-usage() {
-  local cmd="${0##*/}"
-  local line
-
-  cat <<HELP_TEXT_HEADER
-Usage: ${cmd} [OPTIONS] <tests>
-       ${cmd} [-h | -v]
-
-HELP_TEXT_HEADER
-
-  cat <<'HELP_TEXT_BODY'
-  <tests> is the path to a Bats test file, or the path to a directory
-  containing Bats test files (ending with ".bats")
-
-  -c, --count               Count test cases without running any tests
-  -f, --filter <regex>      Only run tests that match the regular expression
-  -F, --formatter <type>    Switch between formatters: pretty (default),
-                              tap (default w/o term), junit
-  -h, --help                Display this help message
-  -j, --jobs <jobs>         Number of parallel jobs (requires GNU parallel)
-  --parallel-preserve-environment
-                            Preserve the current environment for "--jobs"
-                              (run `parallel --record-env` before)
-  --no-tempdir-cleanup      Preserve test output temporary directory
-  -o, --output <dir>        Directory to write report files
-  -p, --pretty              Shorthand for "--formatter pretty"
-  -r, --recursive           Include tests in subdirectories
-  -t, --tap                 Shorthand for "--formatter tap"
-  -T, --timing              Add timing information to tests
-  -v, --version             Display the version number
-
-  For more information, see https://github.com/bats-core/bats-core
-HELP_TEXT_BODY
-}
-
-expand_path() {
-  local path="${1%/}"
-  local dirname="${path%/*}"
-  local result="$2"
-
-  if [[ "$dirname" == "$path" ]]; then
-    dirname="$PWD"
-  else
-    cd "$dirname"
-    dirname="$PWD"
-    cd "$OLDPWD"
-  fi
-  printf -v "$result" '%s/%s' "$dirname" "${path##*/}"
-}
-
-BATS_LIBEXEC="$(dirname "$(bats_readlinkf "${BASH_SOURCE[0]}")")"
-export BATS_CWD="$PWD"
-export BATS_TEST_PATTERN="^[[:blank:]]*@test[[:blank:]]+(.*[^[:blank:]])[[:blank:]]+\{(.*)\$"
-export BATS_TEST_FILTER=
-export PATH="$BATS_LIBEXEC:$PATH"
-export BATS_ROOT_PID=$$
-if [[ -z "$TMPDIR" ]]; then
-  export BATS_TMPDIR="/tmp"
-else
-  export BATS_TMPDIR="${TMPDIR%/}"
-fi
-export BATS_RUN_TMPDIR="$BATS_TMPDIR/bats-run-$BATS_ROOT_PID"
-
-arguments=()
-
-# Unpack single-character options bundled together, e.g. -cr, -pr.
-for arg in "$@"; do
-  if [[ "$arg" =~ ^-[^-]. ]]; then
-    index=1
-    while option="${arg:$((index++)):1}"; do
-      if [[ -z "$option" ]]; then
-        break
-      fi
-      arguments+=("-$option")
-    done
-  else
-    arguments+=("$arg")
-  fi
-  shift
-done
-
-set -- "${arguments[@]}"
-arguments=()
-
-unset flags recursive formatter_flags
-flags=()
-formatter_flags=()
-formatter='tap'
-recursive=
-tempdir_cleanup=1
-output=
-if [[ -z "${CI:-}" && -t 0 && -t 1 ]] && command -v tput >/dev/null; then
-  formatter='pretty'
-fi
-
-while [[ "$#" -ne 0 ]]; do
-  case "$1" in
-  -h | --help)
-    version
-    usage
-    exit 0
-    ;;
-  -v | --version)
-    version
-    exit 0
-    ;;
-  -c | --count)
-    flags+=('-c')
-    ;;
-  -f | --filter)
-    shift
-    flags+=('-f' "$1")
-    ;;
-  -F | --formatter)
-    shift
-    if [[ $1 =~ ^(pretty|junit|tap)$ ]]; then
-      formatter="$1"
-    else
-      printf "Unknown formatter '%s', valid options are pretty, junit, tap\n" "$1"
-      exit 1
-    fi
-    ;;
-  -o | --output)
-    shift
-    output="$1"
-    ;;
-  -p | --pretty)
-    formatter='pretty'
-    ;;
-  -j | --jobs)
-    shift
-    flags+=('-j' "$1")
-    ;;
-  -r | --recursive)
-    recursive=1
-    ;;
-  -t | --tap)
-    formatter='tap'
-    ;;
-  -T | --timing)
-    flags+=('-T')
-    formatter_flags+=('-T')
-    ;;
-  --parallel-preserve-environment)
-    flags+=("--parallel-preserve-environment")
-    # check if parallel's env setup was run before
-    # will print on stderr and return 255 on failure
-    # suppress stdout (on success)
-    parallel --env _ echo ::: 1 >/dev/null || exit 1
-    ;;
-  --no-tempdir-cleanup)
-    tempdir_cleanup=''
-    ;;
-  -*)
-    abort "Bad command line option '$1'"
-    ;;
-  *)
-    arguments+=("$1")
-    ;;
-  esac
-  shift
-done
-
-if [[ -d "$BATS_RUN_TMPDIR" ]]; then
-  printf "Error: BATS_RUN_TMPDIR (%s) already exists\n" "$BATS_RUN_TMPDIR" >&2
-  printf "Reusing old run directories can lead to unexpected results ... aborting!\n" >&2
-  exit 1
-fi
-mkdir -p "$BATS_RUN_TMPDIR"
-if [[ -n "$tempdir_cleanup" ]]; then
-  trap 'rm -rf "$BATS_RUN_TMPDIR"' ERR EXIT
-fi
-
-if [[ "$formatter" != "tap" ]]; then
-  flags+=('-x')
-fi
-if [[ "$formatter" == "junit" ]]; then
-  flags+=('-T')
-  formatter_flags+=('-T')
-fi
-
-if [[ "${#arguments[@]}" -eq 0 ]]; then
-  abort 'Must specify at least one <test>'
-fi
-
-
-if [[ -n "$output" ]]; then
-  if [[ ! -w "${output}" ]]; then
-    abort "Output path ${output} is not writeable"
-  fi
-  export BATS_REPORT_OUTPUT_PATH="$output"
-fi
-
-filenames=()
-for filename in "${arguments[@]}"; do
-  expand_path "$filename" 'filename'
-
-  if [[ -d "$filename" ]]; then
-    shopt -s nullglob
-    if [[ "$recursive" -eq 1 ]]; then
-      while IFS= read -r -d $'\0' file; do
-        filenames+=("$file")
-      done < <(find "$filename" -type f -name '*.bats' -print0 | sort -z)
-    else
-      for suite_filename in "$filename"/*.bats; do
-        filenames+=("$suite_filename")
-      done
-    fi
-    shopt -u nullglob
-  else
-    filenames+=("$filename")
-  fi
-done
-
-# shellcheck source=lib/bats-core/validator.bash
-source "$BATS_ROOT/lib/bats-core/validator.bash"
-
-set -o pipefail execfail
-exec bats-exec-suite "${flags[@]}" "${filenames[@]}" | bats_test_count_validator | "bats-format-${formatter}" "${formatter_flags[@]}"
diff --git a/tools/tests/libexec/bats-core/bats-exec-file b/tools/tests/libexec/bats-core/bats-exec-file
deleted file mode 100755
index efdec103b..000000000
--- a/tools/tests/libexec/bats-core/bats-exec-file
+++ /dev/null
@@ -1,229 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -eET
-
-export flags=()
-num_jobs=1
-filter=''
-extended_syntax=''
-
-while [[ "$#" -ne 0 ]]; do
-  case "$1" in
-  -c) ;;
-
-  -f)
-    shift
-    filter="$1"
-    flags+=('-f' "$filter")
-    ;;
-  -j)
-    shift
-    num_jobs="$1"
-    ;;
-  -T)
-    flags+=('-T')
-    ;;
-  -x)
-    flags+=('-x')
-    extended_syntax=1
-    ;;
-  *)
-    break
-    ;;
-  esac
-  shift
-done
-
-filename=$1
-TESTS_FILE="$2"
-
-if [[ ! -f "$filename" ]]; then
-  printf 'Testfile "%s" not found\n' "$filename" >&2
-  exit 1
-fi
-
-BATS_TEST_FILENAME="$filename"
-
-# shellcheck source=lib/bats-core/preprocessing.bash
-# shellcheck disable=SC2153
-source "$BATS_ROOT/lib/bats-core/preprocessing.bash"
-
-bats_run_setup_file() {
-  # shellcheck source=lib/bats-core/tracing.bash
-  # shellcheck disable=SC2153
-  source "$BATS_ROOT/lib/bats-core/tracing.bash"
-  # shellcheck source=lib/bats-core/test_functions.bash
-  # shellcheck disable=SC2153
-  source "$BATS_ROOT/lib/bats-core/test_functions.bash"
-
-  exec 3<&1
-
-  BATS_STACK_TRACE=()
-  # shellcheck disable=2034
-  BATS_CURRENT_STACK_TRACE=() # used in tracing.bash
-
-  # these are defined only to avoid errors when referencing undefined variables down the line
-  # shellcheck disable=2034
-  BATS_TEST_NAME=      # used in tracing.bash
-  # shellcheck disable=2034
-  BATS_TEST_COMPLETED= # used in tracing.bash
-
-  BATS_SETUP_FILE_COMPLETED=
-  BATS_TEARDOWN_FILE_COMPLETED=
-  # shellcheck disable=2034
-  BATS_ERROR_STATUS= # used in tracing.bash
-  trap 'bats_debug_trap "$BASH_SOURCE"' DEBUG
-  trap 'bats_error_trap' ERR
-  trap 'bats_file_teardown_trap' EXIT
-
-  touch "$BATS_OUT"
-  # get the setup_file/teardown_file functions for this file (if it has them)
-  # shellcheck disable=SC1090
-  source "$BATS_TEST_SOURCE"
-  setup_file >>"$BATS_OUT" 2>&1
-
-  BATS_SETUP_FILE_COMPLETED=1
-}
-
-bats_run_teardown_file() {
-  # avoid running the therdown trap due to errors in teardown_file
-  trap 'bats_file_exit_trap' EXIT
-  local status=0
-  # rely on bats_error_trap to catch failures
-  teardown_file >>"$BATS_OUT" 2>&1
-
-  BATS_TEARDOWN_FILE_COMPLETED=1
-}
-
-bats_file_teardown_trap() {
-  bats_error_trap
-  local status=0
-  bats_run_teardown_file
-
-  bats_file_exit_trap
-}
-
-bats_file_exit_trap() {
-  trap - ERR EXIT
-  if [[ -z "$BATS_SETUP_FILE_COMPLETED" || -z "$BATS_TEARDOWN_FILE_COMPLETED" ]]; then
-    if [[ -z "$BATS_SETUP_FILE_COMPLETED" ]]; then
-      FAILURE_REASON='setup_file'
-    else
-      FAILURE_REASON='teardown_file'
-    fi
-    printf "not ok %d %s\n" "$((test_number + 1))" "$FAILURE_REASON failed" >&3
-    bats_print_stack_trace "${BATS_STACK_TRACE[@]}" >&3
-    bats_print_failed_command >&3
-    while IFS= read -r line; do
-      printf "# %s\n" "$line"
-    done <"$BATS_OUT" >&3
-    if [[ -n "$line" ]]; then
-      printf '# %s\n' "$line"
-    fi
-    rm -rf "$BATS_OUT"
-    status=1
-  fi
-  bats_cleanup_preprocessed_source
-  exit $status
-}
-
-function setup_file() {
-  return 0
-}
-
-function teardown_file() {
-  return 0
-}
-
-bats_run_tests() {
-  status=0
-  tests_to_run=()
-  local line_number=0
-  test_number=''
-  while read -r test_line; do
-    test_file=${test_line%%$'\t'*}
-    test_name=${test_line##*$'\t'}
-    if [[ "$test_file" == "$filename" ]]; then
-      tests_to_run+=("$test_name")
-      # save the first test's number for later iteration
-      # this assumes that tests for a file are stored consecutive in the file!
-      if [[ -z $test_number ]]; then
-        test_number=$line_number
-      fi
-    fi
-    ((++line_number))
-  done <"$TESTS_FILE"
-
-  if [[ "$num_jobs" != 1 ]]; then
-    test_pids_and_numbers=()
-    output_folder="$BATS_RUN_TMPDIR/parallel_output"
-    parallel_log_file="$BATS_RUN_TMPDIR/parallel.log"
-    for test_name in "${tests_to_run[@]}"; do
-      # Only handle non-empty lines
-      if [[ $test_name ]]; then
-        ((++test_number))
-        test_output_dir="$output_folder/$test_number"
-        mkdir -p "$test_output_dir"
-        # parallelize across files using parallel's semaphore mode
-        # use --results to gather output in separate files
-        # use --fg mode to retrieve exit codes (all non zero exit codes seem to mapped to 1), but which requires us to suppress its output
-        # shellcheck disable=SC2154 #bats_parallel_args is inherited from exec-suite
-        parallel --semaphore -j "$num_jobs" "${bats_parallel_args[@]}" --fg -- bats-exec-test "${flags[@]}" "$filename" "$test_name" "$test_number" >"$test_output_dir/stdout" 2>"$test_output_dir/stderr" &
-        pid=$!
-        test_pids_and_numbers+=("$pid,$test_number")
-      fi
-    done
-    for test_pid_and_number in "${test_pids_and_numbers[@]}"; do
-      test_pid=${test_pid_and_number%,*}
-      test_number=${test_pid_and_number##*,}
-      test_output_dir="$output_folder/$test_number"
-      # fetch the return code
-      wait "$test_pid" || status=1
-      # wait for tests to finish in the order they were started and output their results as they come in
-      cat "$test_output_dir/stdout"
-      # additionally output stderr as parallel does by default
-      cat "$test_output_dir/stderr" >&2
-    done
-  else
-    for test_name in "${tests_to_run[@]}"; do
-      # Only handle non-empty lines
-      if [[ $test_name ]]; then
-        ((++test_number))
-        if [[ "${#flags[@]}" -gt 0 ]]; then
-          bats-exec-test "${flags[@]}" "$filename" "$test_name" "$test_number" || status=1
-        else
-          bats-exec-test "$filename" "$test_name" "$test_number" || status=1
-        fi
-      fi
-    done
-  fi
-  export status
-}
-
-if [[ -n "$extended_syntax" ]]; then
-  printf "suite %s\n" "$(basename "$filename")"
-fi
-
-bats_preprocess_source "$filename"
-bats_run_setup_file
-bats_run_tests
-bats_run_teardown_file
-
-exit $status
diff --git a/tools/tests/libexec/bats-core/bats-exec-suite b/tools/tests/libexec/bats-core/bats-exec-suite
deleted file mode 100755
index 5588f07b9..000000000
--- a/tools/tests/libexec/bats-core/bats-exec-suite
+++ /dev/null
@@ -1,132 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-count_only_flag=''
-filter=''
-num_jobs=1
-have_gnu_parallel=
-bats_parallel_args=()
-flags=()
-
-abort() {
-  printf 'Error: %s\n' "$1" >&2
-  exit 1
-}
-
-while [[ "$#" -ne 0 ]]; do
-  case "$1" in
-  -c)
-    count_only_flag=1
-    ;;
-  -f)
-    shift
-    filter="$1"
-    flags+=('-f' "$filter")
-    ;;
-  -j)
-    shift
-    num_jobs="$1"
-    flags+=('-j' "$num_jobs")
-    ;;
-  -T)
-    flags+=('-T')
-    ;;
-  -x)
-    flags+=('-x')
-    ;;
-  --parallel-preserve-environment)
-    bats_parallel_args=("--env" "_")
-    ;;
-  *)
-    break
-    ;;
-  esac
-  shift
-done
-
-if (type -p parallel &>/dev/null); then
-  # shellcheck disable=SC2034
-  have_gnu_parallel=1
-elif [[ "$num_jobs" != 1 ]]; then
-  abort "Cannot execute \"${num_jobs}\" jobs without GNU parallel"
-  exit 1
-fi
-
-trap 'kill 0; exit 1' INT
-
-# create a file that contains all (filtered) tests to run from all files
-TESTS_LIST_FILE="${BATS_RUN_TMPDIR}/test_list_file.txt"
-
-all_tests=()
-for filename in "$@"; do
-  if [[ ! -f "$filename" ]]; then
-    abort "Test file \"${filename}\" does not exist"
-  fi
-
-  test_names=()
-  test_dupes=()
-  while read -r line; do
-    if [[ ! "$line" =~ ^bats_test_function\  ]]; then
-      continue
-    fi
-    line="${line%$'\r'}"
-    line="${line#* }"
-    test_line=$(printf "%s\t%s" "$filename" "$line")
-    all_tests+=("$test_line")
-    printf "%s\n" "$test_line" >>"$TESTS_LIST_FILE"
-    if [[ " ${test_names[*]} " == *" $line "* ]]; then
-      test_dupes+=("$line")
-      continue
-    fi
-    test_names+=("$line")
-  done < <(BATS_TEST_FILTER="$filter" bats-preprocess "$filename")
-
-  if [[ "${#test_dupes[@]}" -ne 0 ]]; then
-    abort "Duplicate test name(s) in file \"${filename}\": ${test_dupes[*]}"
-  fi
-done
-
-test_count="${#all_tests[@]}"
-
-if [[ -n "$count_only_flag" ]]; then
-  printf '%d\n' "${test_count}"
-  exit
-fi
-
-status=0
-printf '1..%d\n' "${test_count}"
-
-# No point on continuing if there's no tests.
-if [[ "${test_count}" == 0 ]]; then
-  exit
-fi
-
-if [[ "$num_jobs" -gt 1 ]]; then
-  # run files in parallel to get the maximum pool of parallel tasks
-  # shellcheck disable=SC2086,SC2068
-  printf "%s\n" "$@" | parallel "${bats_parallel_args[@]}" --keep-order --jobs "$num_jobs" bats-exec-file "${flags[@]}" {} "$TESTS_LIST_FILE" || status=1
-else
-  for filename in "$@"; do
-    bats-exec-file "${flags[@]}" "$filename" "${TESTS_LIST_FILE}" || status=1
-  done
-fi
-
-exit "$status"
diff --git a/tools/tests/libexec/bats-core/bats-exec-test b/tools/tests/libexec/bats-core/bats-exec-test
deleted file mode 100755
index b6be395bc..000000000
--- a/tools/tests/libexec/bats-core/bats-exec-test
+++ /dev/null
@@ -1,185 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -eET
-
-# Variables used in other scripts.
-BATS_COUNT_ONLY=''
-BATS_TEST_FILTER=''
-BATS_ENABLE_TIMING=''
-BATS_EXTENDED_SYNTAX=''
-
-while [[ "$#" -ne 0 ]]; do
-  case "$1" in
-  -c)
-    # shellcheck disable=SC2034
-    BATS_COUNT_ONLY=1
-    ;;
-  -f)
-    shift
-    # shellcheck disable=SC2034
-    BATS_TEST_FILTER="$1"
-    ;;
-  -T)
-    BATS_ENABLE_TIMING='-T'
-    BATS_PERFORM_TEST_CMD+=('-T')
-    ;;
-  -x)
-    # shellcheck disable=SC2034
-    BATS_EXTENDED_SYNTAX='-x'
-    ;;
-  *)
-    break
-    ;;
-  esac
-  shift
-done
-
-BATS_TEST_FILENAME="$1"
-shift
-if [[ -z "$BATS_TEST_FILENAME" ]]; then
-  printf 'usage: bats-exec-test <filename>\n' >&2
-  exit 1
-elif [[ ! -f "$BATS_TEST_FILENAME" ]]; then
-  printf 'bats: %s does not exist\n' "$BATS_TEST_FILENAME" >&2
-  exit 1
-fi
-
-# load the test helper functions like `load` or `run` that are needed to run a (preprocessed) .bats file without bash errors
-# shellcheck source=lib/bats-core/test_functions.bash disable=SC2153
-source "$BATS_ROOT/lib/bats-core/test_functions.bash"
-# shellcheck source=lib/bats-core/tracing.bash disable=SC2153
-source "$BATS_ROOT/lib/bats-core/tracing.bash"
-
-bats_teardown_trap() {
-  bats_error_trap
-  local status=0
-  # mark the start of this function to distinguish where skip is called
-  # parameter 1 will signify the reason why this function was called
-  # this is used to identify when this is called as exit trap function
-  BATS_TEARDOWN_STARTED=${1:-1}
-  teardown >>"$BATS_OUT" 2>&1 || status="$?"
-
-  if [[ $status -eq 0 ]]; then
-    BATS_TEARDOWN_COMPLETED=1
-  elif [[ -n "$BATS_TEST_COMPLETED" ]]; then
-    BATS_ERROR_STATUS="$status"
-  fi
-
-  bats_exit_trap
-}
-
-bats_exit_trap() {
-  local line
-  local status
-  local skipped=''
-  trap - ERR EXIT
-
-  if [[ -n "$BATS_TEST_SKIPPED" ]]; then
-    skipped=' # skip'
-    if [[ "$BATS_TEST_SKIPPED" != '1' ]]; then
-      skipped+=" $BATS_TEST_SKIPPED"
-    fi
-  fi
-
-  BATS_TEST_TIME=""
-  if [[ -z "${skipped}" && -n "$BATS_ENABLE_TIMING" ]]; then
-    BATS_TEST_TIME=" in "$((SECONDS - BATS_TEST_START_TIME))"sec"
-  fi
-
-  if [[ -z "$BATS_TEST_COMPLETED" || -z "$BATS_TEARDOWN_COMPLETED" ]]; then
-    if [[ "$BATS_ERROR_STATUS" -eq 0 ]]; then
-      # For some versions of bash, `$?` may not be set properly for some error
-      # conditions before triggering the EXIT trap directly (see #72 and #81).
-      # Thanks to the `BATS_TEARDOWN_COMPLETED` signal, this will pinpoint such
-      # errors if they happen during `teardown()` when `bats_perform_test` calls
-      # `bats_teardown_trap` directly after the test itself passes.
-      #
-      # If instead the test fails, and the `teardown()` error happens while
-      # `bats_teardown_trap` runs as the EXIT trap, the test will fail with no
-      # output, since there's no way to reach the `bats_exit_trap` call.
-      BATS_STACK_TRACE=("${BATS_CURRENT_STACK_TRACE[@]}")
-      BATS_ERROR_STATUS=1
-    fi
-    printf 'not ok %d %s\n' "$BATS_TEST_NUMBER" "${BATS_TEST_DESCRIPTION}${BATS_TEST_TIME}" >&3
-    bats_print_stack_trace "${BATS_STACK_TRACE[@]}" >&3
-    bats_print_failed_command >&3
-
-    while IFS= read -r line; do
-      printf '# %s\n' "$line"
-    done <"$BATS_OUT" >&3
-    if [[ -n "$line" ]]; then
-      printf '# %s\n' "$line"
-    fi
-    status=1
-  else
-    printf 'ok %d %s%s\n' "$BATS_TEST_NUMBER" "${BATS_TEST_DESCRIPTION}${BATS_TEST_TIME}" \
-      "$skipped" >&3
-    status=0
-  fi
-
-  rm -f "$BATS_OUT"
-  bats_cleanup_preprocessed_source
-  exit "$status"
-}
-
-bats_perform_test() {
-  BATS_TEST_NAME="$1"
-  BATS_TEST_NUMBER="$2"
-
-  if ! declare -F "$BATS_TEST_NAME" &>/dev/null; then
-    printf "bats: unknown test name \`%s'\n" "$BATS_TEST_NAME" >&2
-    exit 1
-  fi
-
-  # Some versions of Bash will reset BASH_LINENO to the first line of the
-  # function when the ERR trap fires. All versions of Bash appear to reset it
-  # on an unbound variable access error. bats_debug_trap will fire both before
-  # the offending line is executed, and when the error is triggered.
-  # Consequently, we use `BATS_CURRENT_STACK_TRACE` recorded by the
-  # first call to bats_debug_trap, _before_ the ERR trap or unbound variable
-  # access fires.
-  BATS_STACK_TRACE=()
-  BATS_CURRENT_STACK_TRACE=()
-
-  BATS_TEST_COMPLETED=
-  BATS_TEST_SKIPPED=
-  BATS_TEARDOWN_COMPLETED=
-  BATS_ERROR_STATUS=
-  trap 'bats_debug_trap "$BASH_SOURCE"' DEBUG
-  trap 'bats_error_trap' ERR
-  # mark this call as trap call
-  trap 'bats_teardown_trap as-exit-trap' EXIT
-
-  "$BATS_TEST_NAME" >>"$BATS_OUT" 2>&1
-
-  BATS_TEST_COMPLETED=1
-  trap 'bats_exit_trap' EXIT
-  bats_teardown_trap
-}
-
-# shellcheck source=lib/bats-core/preprocessing.bash
-source "$BATS_ROOT/lib/bats-core/preprocessing.bash"
-
-exec 3<&1
-
-# Run the given test.
-bats_preprocess_source
-bats_evaluate_preprocessed_source
-bats_perform_test "$@"
diff --git a/tools/tests/libexec/bats-core/bats-format-junit b/tools/tests/libexec/bats-core/bats-format-junit
deleted file mode 100755
index 4948f5603..000000000
--- a/tools/tests/libexec/bats-core/bats-format-junit
+++ /dev/null
@@ -1,179 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -euo pipefail
-
-index=0
-
-init_suite() {
-  count=0
-  failures=0
-  skipped=0
-  suitetest_exec_time=0
-  name=""
-  _buffer=""
-  _buffer_log=""
-}
-
-init_suite
-
-host() {
-  local hostname="${HOST:-}"
-  [[ -z "$hostname" ]] && hostname="${HOSTNAME:-}"
-  [[ -z "$hostname" ]] && hostname="$(uname -n)"
-  [[ -z "$hostname" ]] && hostname="$(hostname -f)"
-
-  echo "$hostname"
-}
-
-header() {
-  timestamp=$(date -u +"%Y-%m-%dT%H:%M:%S")
-  printf "\
-<?xml version=\"1.0\" encoding=\"UTF-8\"?>
-<testsuite name=\"%s\" tests=\"%s\" failures=\"%s\" errors=\"0\" skipped=\"%s\" time=\"%s\" timestamp=\"%s\" hostname=\"%s\">
-  <properties>" "${class}" "${count}" "${failures}" "${skipped}" "${suitetest_exec_time}" "${timestamp}" "$(host)"
-  while IFS= read -r line; do
-    name="$(echo "$line" | cut -d'=' -f1)"
-    value="$(echo "$line" | cut -d'=' -f2-)"
-    printf "\n    <property name=\"%s\" value=\"%s\"/>" "${name}" "${value}"
-  done < <(env | grep "^BATS_")
-  printf "\n  </properties>"
-}
-
-footer() {
-  printf "  <system-out></system-out>\n"
-  printf "  <system-err></system-err>\n"
-  printf "</testsuite>\n"
-}
-
-pass() {
-  printf "\n    <testcase classname=\"%s\" name=\"%s\" time=\"%s\"/>" "${class}" "${name}" "${test_exec_time}"
-}
-
-fail() {
-  printf "\n    <testcase classname=\"%s\" name=\"%s\" time=\"%s\">
-        <failure type=\"failure\"><![CDATA[" "${class}" "${name}" "${test_exec_time}"
-
-  echo -n "${1}"
-
-  printf "]]></failure>
-    </testcase>\n"
-}
-
-skip() {
-  # shellcheck disable=SC2183
-  printf "\n    <testcase classname=\"%s\" name=\"%s\" time=\"%s\">
-        <skipped>%s</skipped>
-    </testcase>\n" "${class}" "${name}" "${test_exec_time}" "$1"
-}
-
-buffer() {
-  _buffer="${_buffer}$("$@")"
-}
-
-flush() {
-  echo "${_buffer}"
-  _buffer=""
-}
-
-log() {
-  _buffer_log="${_buffer_log}
-$1"
-}
-
-flush_log() {
-  if [[ -n "${_buffer_log}" ]]; then
-    buffer fail "${_buffer_log}"
-    _buffer_log=""
-  fi
-}
-
-finish_suite() {
-  [[ ${count} -gt 0 ]] && {
-    (
-      flush_log
-      header
-      flush
-      footer
-    ) >"$(output_path)"
-  }
-  init_suite
-}
-
-output_path() {
-  path="TestReport-${class}.xml"
-  if [[ -n "${BATS_REPORT_OUTPUT_PATH:-}" ]]; then
-    path="${BATS_REPORT_OUTPUT_PATH%/}/${path}"
-  fi
-  echo "$path"
-}
-
-trap finish_suite EXIT
-
-while IFS= read -r line; do
-  case "$line" in
-  "suite "*)
-    flush_log
-    finish_suite
-    suite_expr="suite (.*)"
-    if [[ "$line" =~ $suite_expr ]]; then
-      class="${BASH_REMATCH[1]}"
-    fi
-    ;;
-  "begin "*)
-    flush_log
-    ((index += 1))
-    name="${line#* $index }"
-    ;;
-  "ok "*)
-    echo "$line"
-    ((count += 1))
-    expr_ok="ok $index .* in ([0-9]+)sec"
-    expr_skip="ok $index .* # skip[ ]?(.*)"
-    if [[ "$line" =~ $expr_skip ]]; then
-      ((skipped += 1))
-      test_exec_time=0
-      buffer skip "${BASH_REMATCH[1]}"
-    elif [[ "$line" =~ $expr_ok ]]; then
-      test_exec_time="${BASH_REMATCH[1]}"
-      suitetest_exec_time=$((suitetest_exec_time + test_exec_time))
-      buffer pass
-    else
-      log "Wrong output format: ${line}"
-      ((failures += 1))
-    fi
-    ;;
-  "not ok "*)
-    echo "$line"
-    ((count += 1))
-    ((failures += 1))
-    expr_notok="not ok $index .* in ([0-9]+)sec"
-    if [[ "$line" =~ $expr_notok ]]; then
-      test_exec_time="${BASH_REMATCH[1]}"
-      suitetest_exec_time=$((suitetest_exec_time + test_exec_time))
-    fi
-    ;;
-  "# "*)
-    log "${line:2}"
-    ;;
-  *)
-    echo "$line"
-    ;;
-  esac
-done
diff --git a/tools/tests/libexec/bats-core/bats-format-pretty b/tools/tests/libexec/bats-core/bats-format-pretty
deleted file mode 100755
index 9c93ad3f9..000000000
--- a/tools/tests/libexec/bats-core/bats-format-pretty
+++ /dev/null
@@ -1,265 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-while [[ "$#" -ne 0 ]]; do
-  case "$1" in
-  -T)
-    BATS_ENABLE_TIMING="-T"
-    ;;
-  esac
-  shift
-done
-
-header_pattern='[0-9]+\.\.[0-9]+'
-IFS= read -r header
-
-update_count_column_width() {
-  count_column_width=$((${#count} * 2 + 2))
-  if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-    # additional space for ' in %s sec'
-    count_column_width=$((count_column_width + ${#SECONDS} + 8))
-  fi
-  # also update dependent value
-  update_count_column_left
-}
-
-update_screen_width() {
-  screen_width="$(tput cols)"
-  # also update dependent value
-  update_count_column_left
-}
-
-update_count_column_left() {
-  count_column_left=$((screen_width - count_column_width))
-}
-
-if [[ "$header" =~ $header_pattern ]]; then
-  count="${header:3}"
-  index=0
-  passed=0
-  failures=0
-  skipped=0
-  name=
-  update_count_column_width
-else
-  # If the first line isn't a TAP plan, print it and pass the rest through
-  printf '%s\n' "$header"
-  exec cat
-fi
-
-trap update_screen_width WINCH
-update_screen_width
-
-begin() {
-  go_to_column 0
-  update_count_column_width
-  buffer_with_truncation $((count_column_left - 1)) '   %s' "$name"
-  clear_to_end_of_line
-  go_to_column $count_column_left
-  if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-    buffer "%${#count}s/${count} in %s sec" "$index" "$SECONDS"
-  else
-    buffer "%${#count}s/${count}" "$index"
-  fi
-  go_to_column 1
-}
-
-pass() {
-  go_to_column 0
-  if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-    buffer ' ✓ %s [%s]' "$name" "$1"
-  else
-    buffer ' ✓ %s ' "$name"
-  fi
-  advance
-}
-
-skip() {
-  local reason="$1"
-  if [[ -n "$reason" ]]; then
-    reason=": $reason"
-  fi
-  go_to_column 0
-  buffer ' - %s (skipped%s)' "$name" "$reason"
-  advance
-}
-
-fail() {
-  go_to_column 0
-  set_color 1 bold
-  if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-    buffer ' ✗ %s [%s]' "$name" "$1"
-  else
-    buffer ' ✗ %s' "$name"
-  fi
-  advance
-}
-
-log() {
-  set_color 1
-  buffer '   %s\n' "$1"
-  clear_color
-}
-
-summary() {
-  buffer '\n%d test' "$count"
-  if [[ "$count" -ne 1 ]]; then
-    buffer 's'
-  fi
-
-  buffer ', %d failure' "$failures"
-  if [[ "$failures" -ne 1 ]]; then
-    buffer 's'
-  fi
-
-  if [[ "$skipped" -gt 0 ]]; then
-    buffer ', %d skipped' "$skipped"
-  fi
-
-  not_run=$((count - passed - failures - skipped))
-  if [[ "$not_run" -gt 0 ]]; then
-    buffer ', %d not run' "$not_run"
-  fi
-
-  if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-    buffer " in $SECONDS seconds"
-  fi
-
-  buffer '\n'
-}
-
-buffer_with_truncation() {
-  local width="$1"
-  shift
-  local string
-
-  # shellcheck disable=SC2059
-  printf -v 'string' -- "$@"
-
-  if [[ "${#string}" -gt "$width" ]]; then
-    buffer '%s...' "${string:0:$((width - 4))}"
-  else
-    buffer '%s' "$string"
-  fi
-}
-
-go_to_column() {
-  local column="$1"
-  buffer '\x1B[%dG' $((column + 1))
-}
-
-clear_to_end_of_line() {
-  buffer '\x1B[K'
-}
-
-advance() {
-  clear_to_end_of_line
-  buffer '\n'
-  clear_color
-}
-
-set_color() {
-  local color="$1"
-  local weight=22
-
-  if [[ "$2" == 'bold' ]]; then
-    weight=1
-  fi
-  buffer '\x1B[%d;%dm' "$((30 + color))" "$weight"
-}
-
-clear_color() {
-  buffer '\x1B[0m'
-}
-
-_buffer=
-
-buffer() {
-  local content
-  # shellcheck disable=SC2059
-  printf -v content -- "$@"
-  _buffer+="$content"
-}
-
-flush() {
-  printf '%s' "$_buffer"
-  [[ -n "$BATS_REPORT_OUTPUT_PATH" ]] && printf '%s' "$_buffer" | tee "$BATS_REPORT_OUTPUT_PATH/report.tap" >/dev/null
-  _buffer=
-}
-
-finish() {
-  flush
-  printf '\n'
-}
-
-trap finish EXIT
-
-timing_expr="in (([0-9]+sec))$"
-
-while IFS= read -r line; do
-  case "$line" in
-  'begin '*)
-    ((++index))
-    name="${line#* $index }"
-    begin
-    flush
-    ;;
-  'ok '*)
-    skip_expr="ok $index (.*) # skip ?(([[:print:]]*))?"
-    if [[ "$line" =~ $skip_expr ]]; then
-      ((++skipped))
-      skip "${BASH_REMATCH[2]}"
-    else
-      ((++passed))
-      if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-        if [[ "$line" =~ $timing_expr ]]; then
-          pass "${BASH_REMATCH[2]}"
-        else
-          echo "Could not match output line to timing regex: $line" >&2
-          exit 1
-        fi
-      else
-        pass
-      fi
-    fi
-    ;;
-  'not ok '*)
-    ((++failures))
-    if [[ -n "$BATS_ENABLE_TIMING" ]]; then
-      if [[ "$line" =~ $timing_expr ]]; then
-        fail "${BASH_REMATCH[2]}"
-      else
-        echo "Could not match failure line to timing regex: $line" >&2
-        exit 1
-      fi
-    else
-      fail
-    fi
-    ;;
-  '# '*)
-    log "${line:2}"
-    ;;
-  'suite '*) ;;
-
-  esac
-done
-
-summary
diff --git a/tools/tests/libexec/bats-core/bats-format-tap b/tools/tests/libexec/bats-core/bats-format-tap
deleted file mode 100755
index 1a37ed881..000000000
--- a/tools/tests/libexec/bats-core/bats-format-tap
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-cat
diff --git a/tools/tests/libexec/bats-core/bats-preprocess b/tools/tests/libexec/bats-core/bats-preprocess
deleted file mode 100755
index 1e6d7a7dd..000000000
--- a/tools/tests/libexec/bats-core/bats-preprocess
+++ /dev/null
@@ -1,75 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-bats_encode_test_name() {
-  local name="$1"
-  local result='test_'
-  local hex_code
-
-  if [[ ! "$name" =~ [^[:alnum:]\ _-] ]]; then
-    name="${name//_/-5f}"
-    name="${name//-/-2d}"
-    name="${name// /_}"
-    result+="$name"
-  else
-    local length="${#name}"
-    local char i
-
-    for ((i = 0; i < length; i++)); do
-      char="${name:$i:1}"
-      if [[ "$char" == ' ' ]]; then
-        result+='_'
-      elif [[ "$char" =~ [[:alnum:]] ]]; then
-        result+="$char"
-      else
-        printf -v 'hex_code' -- '-%02x' \'"$char"
-        result+="$hex_code"
-      fi
-    done
-  fi
-
-  printf -v "$2" '%s' "$result"
-}
-
-test_file="$1"
-tests=()
-{
-  while IFS= read -r line; do
-    line="${line//$'\r'/}"
-    if [[ "$line" =~ $BATS_TEST_PATTERN ]]; then
-      name="${BASH_REMATCH[1]#[\'\"]}"
-      name="${name%[\'\"]}"
-      body="${BASH_REMATCH[2]}"
-      bats_encode_test_name "$name" 'encoded_name'
-      printf '%s() { bats_test_begin "%s"; %s\n' "${encoded_name:?}" "$name" "$body" || :
-
-      if [[ -z "$BATS_TEST_FILTER" || "$name" =~ $BATS_TEST_FILTER ]]; then
-        tests+=("$encoded_name")
-      fi
-    else
-      printf '%s\n' "$line"
-    fi
-  done
-} <<<"$(<"$test_file")"$'\n'
-
-for test_name in "${tests[@]}"; do
-  printf 'bats_test_function %s\n' "$test_name"
-done
diff --git a/tools/tests/share/man/man1/bats.1 b/tools/tests/share/man/man1/bats.1
deleted file mode 100644
index 6f673824a..000000000
--- a/tools/tests/share/man/man1/bats.1
+++ /dev/null
@@ -1,94 +0,0 @@
-.\" generated with Ronn-NG/v0.9.0
-.\" http://github.com/apjanke/ronn-ng/tree/0.9.0
-.TH "BATS" "1" "April 2020" "bats-core" "Bash Automated Testing System"
-.SH "NAME"
-\fBbats\fR \- Bash Automated Testing System
-.SH "SYNOPSIS"
-Usage: bats [OPTIONS] \fItests\fR bats [\-h | \-v]
-.P
-\fItests\fR is the path to a Bats test file, or the path to a directory containing Bats test files (ending with "\.bats")
-.SH "DESCRIPTION"
-Bats is a TAP\-compliant testing framework for Bash\. It provides a simple way to verify that the UNIX programs you write behave as expected\.
-.P
-A Bats test file is a Bash script with special syntax for defining test cases\. Under the hood, each test case is just a function with a description\.
-.P
-Test cases consist of standard shell commands\. Bats makes use of Bash\'s \fBerrexit\fR (\fBset \-e\fR) option when running test cases\. If every command in the test case exits with a \fB0\fR status code (success), the test passes\. In this way, each line is an assertion of truth\.
-.P
-See \fBbats\fR(7) for more information on writing Bats tests\.
-.SH "RUNNING TESTS"
-To run your tests, invoke the \fBbats\fR interpreter with a path to a test file\. The file\'s test cases are run sequentially and in isolation\. If all the test cases pass, \fBbats\fR exits with a \fB0\fR status code\. If there are any failures, \fBbats\fR exits with a \fB1\fR status code\.
-.P
-You can invoke the \fBbats\fR interpreter with multiple test file arguments, or with a path to a directory containing multiple \fB\.bats\fR files\. Bats will run each test file individually and aggregate the results\. If any test case fails, \fBbats\fR exits with a \fB1\fR status code\.
-.SH "OPTIONS"
-.TP
-\fB\-c\fR, \fB\-\-count\fR
-Count the number of test cases without running any tests
-.TP
-\fB\-f\fR, \fB\-\-filter <regex>\fR
-Filter test cases by names matching the regular expression
-.TP
-\fB\-F\fR, \fB\-\-formatter <type>\fR
-Switch between formatters: pretty (default), tap (default w/o term), junit
-.TP
-\fB\-h\fR, \fB\-\-help\fR
-Display this help message
-.TP
-\fB\-j\fR, \fB\-\-jobs <jobs>\fR
-Display this help message
-.TP
-\fB\-\-parallel\-preserve\-environment\fR
-Preserve the current environment for "\-\-jobs" (run \fBparallel \-\-record\-env\fR before)
-.TP
-\fB\-\-no\-tempdir\-cleanup\fR
-Preserve test output temporary directory
-.TP
-\fB\-o\fR, \fB\-\-output <dir>\fR
-Directory to write report files
-.TP
-\fB\-p\fR, \fB\-\-pretty\fR
-Shorthand for "\-\-formatter pretty"
-.TP
-\fB\-r\fR, \fB\-\-recursive\fR
-Include tests in subdirectories
-.TP
-\fB\-t\fR, \fB\-\-tap\fR
-Shorthand for "\-\-formatter tap"
-.TP
-\fB\-T\fR, \fB\-\-timing\fR
-Add timing information to tests
-.TP
-\fB\-v\fR, \fB\-\-version\fR
-Display the version number
-.SH "OUTPUT"
-When you run Bats from a terminal, you\'ll see output as each test is performed, with a check\-mark next to the test\'s name if it passes or an "X" if it fails\.
-.IP "" 4
-.nf
-$ bats addition\.bats
- ✓ addition using bc
- ✓ addition using dc
-
-2 tests, 0 failures
-.fi
-.IP "" 0
-.P
-If Bats is not connected to a terminal\-\-in other words, if you run it from a continuous integration system or redirect its output to a file\-\-the results are displayed in human\-readable, machine\-parsable TAP format\. You can force TAP output from a terminal by invoking Bats with the \fB\-\-tap\fR option\.
-.IP "" 4
-.nf
-$ bats \-\-tap addition\.bats
-1\.\.2
-ok 1 addition using bc
-ok 2 addition using dc
-.fi
-.IP "" 0
-.SH "EXIT STATUS"
-The \fBbats\fR interpreter exits with a value of \fB0\fR if all test cases pass, or \fB1\fR if one or more test cases fail\.
-.SH "SEE ALSO"
-Bats wiki: \fIhttps://github\.com/bats\-core/bats\-core/wiki/\fR
-.P
-\fBbash\fR(1), \fBbats\fR(7)
-.SH "COPYRIGHT"
-(c) 2017\-2018 bats\-core organization
-.br
-(c) 2011\-2016 Sam Stephenson
-.P
-Bats is released under the terms of an MIT\-style license\.
diff --git a/tools/tests/share/man/man7/bats.7 b/tools/tests/share/man/man7/bats.7
deleted file mode 100644
index bd702c20e..000000000
--- a/tools/tests/share/man/man7/bats.7
+++ /dev/null
@@ -1,178 +0,0 @@
-.\" generated with Ronn/v0.7.3
-.\" http://github.com/rtomayko/ronn/tree/0.7.3
-.
-.TH "BATS" "7" "May 2019" "bats-core" "Bash Automated Testing System"
-.
-.SH "NAME"
-\fBbats\fR \- Bats test file format
-.
-.SH "DESCRIPTION"
-A Bats test file is a Bash script with special syntax for defining test cases\. Under the hood, each test case is just a function with a description\.
-.
-.IP "" 4
-.
-.nf
-
-#!/usr/bin/env bats
-
-@test "addition using bc" {
-  result="$(echo 2+2 | bc)"
-  [ "$result" \-eq 4 ]
-}
-
-@test "addition using dc" {
-  result="$(echo 2 2+p | dc)"
-  [ "$result" \-eq 4 ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.P
-Each Bats test file is evaluated n+1 times, where \fIn\fR is the number of test cases in the file\. The first run counts the number of test cases, then iterates over the test cases and executes each one in its own process\.
-.
-.SH "THE RUN HELPER"
-Many Bats tests need to run a command and then make assertions about its exit status and output\. Bats includes a \fBrun\fR helper that invokes its arguments as a command, saves the exit status and output into special global variables, and then returns with a \fB0\fR status code so you can continue to make assertions in your test case\.
-.
-.P
-For example, let\'s say you\'re testing that the \fBfoo\fR command, when passed a nonexistent filename, exits with a \fB1\fR status code and prints an error message\.
-.
-.IP "" 4
-.
-.nf
-
-@test "invoking foo with a nonexistent file prints an error" {
-  run foo nonexistent_filename
-  [ "$status" \-eq 1 ]
-  [ "$output" = "foo: no such file \'nonexistent_filename\'" ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.P
-The \fB$status\fR variable contains the status code of the command, and the \fB$output\fR variable contains the combined contents of the command\'s standard output and standard error streams\.
-.
-.P
-A third special variable, the \fB$lines\fR array, is available for easily accessing individual lines of output\. For example, if you want to test that invoking \fBfoo\fR without any arguments prints usage information on the first line:
-.
-.IP "" 4
-.
-.nf
-
-@test "invoking foo without arguments prints usage" {
-  run foo
-  [ "$status" \-eq 1 ]
-  [ "${lines[0]}" = "usage: foo <filename>" ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.SH "THE LOAD COMMAND"
-You may want to share common code across multiple test files\. Bats includes a convenient \fBload\fR command for sourcing a Bash source file relative to the location of the current test file\. For example, if you have a Bats test in \fBtest/foo\.bats\fR, the command
-.
-.IP "" 4
-.
-.nf
-
-load test_helper
-.
-.fi
-.
-.IP "" 0
-.
-.P
-will source the script \fBtest/test_helper\.bash\fR in your test file\. This can be useful for sharing functions to set up your environment or load fixtures\.
-.
-.SH "THE SKIP COMMAND"
-Tests can be skipped by using the \fBskip\fR command at the point in a test you wish to skip\.
-.
-.IP "" 4
-.
-.nf
-
-@test "A test I don\'t want to execute for now" {
-  skip
-  run foo
-  [ "$status" \-eq 0 ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.P
-Optionally, you may include a reason for skipping:
-.
-.IP "" 4
-.
-.nf
-
-@test "A test I don\'t want to execute for now" {
-  skip "This command will return zero soon, but not now"
-  run foo
-  [ "$status" \-eq 0 ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.P
-Or you can skip conditionally:
-.
-.IP "" 4
-.
-.nf
-
-@test "A test which should run" {
-  if [ foo != bar ]; then
-    skip "foo isn\'t bar"
-  fi
-
-  run foo
-  [ "$status" \-eq 0 ]
-}
-.
-.fi
-.
-.IP "" 0
-.
-.SH "SETUP AND TEARDOWN FUNCTIONS"
-You can define special \fBsetup\fR and \fBteardown\fR functions which run before and after each test case, respectively\. Use these to load fixtures, set up your environment, and clean up when you\'re done\.
-.
-.SH "CODE OUTSIDE OF TEST CASES"
-You can include code in your test file outside of \fB@test\fR functions\. For example, this may be useful if you want to check for dependencies and fail immediately if they\'re not present\. However, any output that you print in code outside of \fB@test\fR, \fBsetup\fR or \fBteardown\fR functions must be redirected to \fBstderr\fR (\fB>&2\fR)\. Otherwise, the output may cause Bats to fail by polluting the TAP stream on \fBstdout\fR\.
-.
-.SH "SPECIAL VARIABLES"
-There are several global variables you can use to introspect on Bats tests:
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_FILENAME\fR is the fully expanded path to the Bats test file\.
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_DIRNAME\fR is the directory in which the Bats test file is located\.
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_NAMES\fR is an array of function names for each test case\.
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_NAME\fR is the name of the function containing the current test case\.
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_DESCRIPTION\fR is the description of the current test case\.
-.
-.IP "\(bu" 4
-\fB$BATS_TEST_NUMBER\fR is the (1\-based) index of the current test case in the test file\.
-.
-.IP "\(bu" 4
-\fB$BATS_TMPDIR\fR is the location to a directory that may be used to store temporary files\.
-.
-.IP "" 0
-.
-.SH "SEE ALSO"
-\fBbash\fR(1), \fBbats\fR(1)