diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md new file mode 100644 index 00000000..4fd8b640 --- /dev/null +++ b/DEPLOYMENT.md @@ -0,0 +1,56 @@ +# Deployment Guide + +## Set-up the environment(s) + +1. Clone the repository and checkout a release branch using the below command: + +``` +git clone --origin upstream --branch release/1.0.0 https://github.com/awslabs/mlops-modules +``` +The release version can be replaced with the version of interest. + +2. Move into the `mlops-modules` repository: +``` +cd mlops-modules +``` +3. Create and activate a Virtual environment +``` +python3 -m venv .venv && source .venv/bin/activate +``` +4. Install the requirements +``` +pip install -r ./requirements.txt +``` +5. Set environment variables + +Replace the values below with your AWS account id and Administrator IAM Role. +``` +export PRIMARY_ACCOUNT=XXXXXXXXXXXX +export ADMIN_ROLE_ARN=arn:aws:iam::XXXXXXXXXXXX:role/XXXXX +``` + +5. Bbootstrap the CDK environment (one time per region) with CDK V2. Asuming you are deploying in `us-east-1`: +``` +cdk bootstrap aws://${PRIMARY_ACCOUNT}/us-east-1 +``` +6. Bootstrap AWS Account(s) + +Assuming that you will be using a single account, follow the guide [here](https://seed-farmer.readthedocs.io/en/latest/bootstrapping.html#) to bootstrap your account(s) to function as a toolchain and target account. + +Following is the command to bootstrap your existing account to a toolchain and target account. +``` +seedfarmer bootstrap toolchain --project mlops --trusted-principal ${ADMIN_ROLE_ARN} --as-target +``` + +## Deployment + +Note: if you are deploying into a region different from `us-east-1`, change the `regionMappings` in `manifests/deployment.yaml`. +``` +seedfarmer apply manifests/deployment.yaml +``` +## Clean-up + +Do destroy all modules, run: +``` +seedfarmer destroy mlops +``` diff --git a/README.md b/README.md index 20e86262..023b917f 100644 --- a/README.md +++ b/README.md @@ -6,24 +6,29 @@ The modules in this repository are decoupled from each other and can be aggregat ## General Information -The modules in this repository are / must be generic for reuse without affiliation to any one particular project in Machine Learning Operations domain. +The modules in this repository are / must be generic for reuse without affiliation to any one particular project in Machine Learning and Foundation Model Operations domain. All modules in this repository adhere to the module structure defined in the the [SeedFarmer Guide](https://seed-farmer.readthedocs.io/en/latest) - [Project Structure](https://seed-farmer.readthedocs.io/en/latest/project_development.html) - [Module Development](https://seed-farmer.readthedocs.io/en/latest/module_development.html) - [Module Manifest Guide](https://seed-farmer.readthedocs.io/en/latest/manifests.html) +- [Seed-Farmer Workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/dfd2f6b2-3923-4d79-80bd-7db6c4842122/en-US) + +## Deployment + +See deployment steps in the [Deployment Guide](DEPLOYMENT.md). ## Modules ### SageMaker Modules -| Type | Description | -|---------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md) | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config | -| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md) | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group | -| [SageMaker Project Templates in Service Catalog Module](modules/sagemaker/sagemaker-templates-service-catalog/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog | -| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md) | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace | +| Type | Description | +|---------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [SageMaker Studio Module](modules/sagemaker/sagemaker-studio/README.md) | Provisions secure SageMaker Studio Domain environment, creates example User Profiles for Data Scientist and Lead Data Scientist linked to IAM Roles, and adds lifecycle config | +| [SageMaker Endpoint Module](modules/sagemaker/sagemaker-endpoint/README.md) | Creates SageMaker real-time inference endpoint for the specified model package or latest approved model from the model package group | +| [SageMaker Project Templates via Service Catalog Module](modules/sagemaker/sagemaker-templates-service-catalog/README.md) | Provisions SageMaker Project Templates for an organization. The templates are available using SageMaker Studio Classic or Service Catalog | +| [SageMaker Notebook Instance Module](modules/sagemaker/sagemaker-notebook/README.md) | Creates secure SageMaker Notebook Instance for the Data Scientist, clones the source code to the workspace | ### Mlflow Modules diff --git a/modules/sagemaker/sagemaker-templates-service-catalog/README.md b/modules/sagemaker/sagemaker-templates-service-catalog/README.md index a710c3d0..e85531d8 100644 --- a/modules/sagemaker/sagemaker-templates-service-catalog/README.md +++ b/modules/sagemaker/sagemaker-templates-service-catalog/README.md @@ -6,7 +6,7 @@ The templates are registered in Service Catalog and available via SageMaker Stud ### Architecture -![SageMaker Templates Module Architecture](docs/_static/sagemaker-templates-module-architecture.png "SageMaker Templates Module Architecture") +![SageMaker Templates via Service Catalog Module Architecture](docs/_static/sagemaker-templates-service-catalog-module-architecture.png "SageMaker Templates in Service Catalog Module Architecture") ### Project Templates @@ -14,9 +14,10 @@ The module contains ogranizational SageMaker Project Templates vended as Service #### Basic Multi-Account Template -This project template contains an example of basic multi-account template from [AWS Enterprise MLOps Framework](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/README.md#sagemaker-project-stack). +This project template contains basic multi-account template from [AWS Enterprise MLOps Framework](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/README.md#sagemaker-project-stack). +The template contains SageMaker Pipeline to train a model on Abalone dataset using XGBoost, perform evaluation, and an example CI/CD pipeline to deploy the model endpoints to multiple AWS accounts. -TODO: add detailed description and architecture diagram. +![Basic Multi-Account Template](https://github.com/aws-samples/aws-enterprise-mlops-framework/blob/main/mlops-multi-account-cdk/mlops-sm-project-template/diagrams/mlops-sm-project-general-architecture.jpg) ## Inputs and outputs: ### Required inputs: diff --git a/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.png b/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.png new file mode 100644 index 00000000..6cb15d8e Binary files /dev/null and b/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.png differ diff --git a/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.xml b/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.xml new file mode 100644 index 00000000..af46100f --- /dev/null +++ b/modules/sagemaker/sagemaker-templates-service-catalog/docs/_static/sagemaker-templates-service-catalog-module-architecture.xml @@ -0,0 +1 @@ +7V1bd5u6Ev41eSQLAeLyaDtJd1bbs3Lq7u7T89IlQNg0GPkAjuP++jPiYnMLcWI7hkRt2qBBiJEYaT59GsGFOlk8forIcv6VuTS4UGT38UK9ulAUS9fgfy7YZAId54JZ5LuZCO0EU/8PzYVyLl35Lo0rGRPGgsRfVoUOC0PqJBUZiSK2rmbzWFC965LMaEMwdUjQlP7ju8k8k5qKsZP/Rf3ZvLgz0q3szIIUmfOaxHPisnVJpF5fqJOIsSQ7WjxOaMDbrmiX/LqQPib8zK37gwSrXC1V25bwjcZsFTn0isZO5C8TFsFFUS7Mcv+6UEeFFjTySeD/IYnPQumBRjH8znI95FlI3mRRS8H5Pad0QcLEd65IQiYsTIgf0mif0rOrk8gPZ1/8hEYkyJ5dQsOkUutlxJY0SnKrmScJf96jC+UGfiA7C9hscxlTZxX5yeaSLMgfFl669AFOe2wVuqkGkHB9MovIQnrw49VWM5ATRcHYwrrkqK4naY7iSLZmWpCUDU83ie1Y2SO+yXS+/XbbaNYXaQVW7M9CyQ/jJdgpb8sbhy2WLISax5AwNWLKNvYkrGuKpBFkSBY2sURtz7V17Gk2dY7bNPEmTuhCWvAuC88DJLKGVQtbhqQppippno4lU7ZkiVDd8CxXpZZmlRsFDtotoTjbYpvFqdzc201/2+uiql28xODRQAweCYP/cAbP7N/cWSlyQGzwl2m9cnMFw3D84Ptmmbf5jEL5viNt2y7LV/QU+2Y9+9+3OKCz3//99+rz90/TL799CaGt7W37VpxsCqcW8VahvAT5Qh2v52CY0yVJn/IavDjI5smCq4Xg0PODYAJtDDW5CkEDELkknqeX8/OZPmD6CX18svfmosj1/l6CeWY3L3nqOKYJr/OPcg9SS5X4RNmCJtEGMud3UVQzuzi3C8XMu/N656exnDfrvOyj9Vq/n23L3j3pPQeZruYfwNjTrb4YkgYwJJ28UTQPI8tCnuSpjg6NokGj2LIsyUjFukxsS/G0pxplq8N6vb5cq5cs4lWE4ixeU14JRZFgUJDiDdj1oxTCDdWEj33HrGGlxWd8miLFSbRyklVEuR62RilxTEnWKJY0Cg/c1C2wRRfrnm1phm3YJ61gNx7+a/M5uCYRviVX9/Lq8/LPHf70Nv1Txo6DVeRIrqIhSbMcFUzR5Q2jGobhYE3RvVN4R65+5iBf7SznBKaDwT6uUtnNpNpcJXVhOpgnWZTM2YyFJLjeScdVZ1pynPTRT/5TOv7Js1ziPHX1mF+RJjalxB1UAfwRb7dU9psmySafGZNVwkC00+QLY8uKI+YKP+eGyz3pCReQZUxINKOdviJz0kfy7BENwPYeaK2EE3lpRRu0l1aeHHXP66XHAQnv/wUO7fbqJZDpuJVpjvIu8hxZ16mkmjr37VSWTFXWJN2hngYuTCZYPrwaRXc4XT10ZNgUernkeABPNCIjiRjYlUxEXRWcs4JV5S29p44RNnRFlgzNBn0MmYJ/sFzJtmwP67bu2NppvadAhS2osHM8+Tu8D9k6vOPUpwASepDwKa0PtdFnSepHM5FdCEb/TOGiKY0efDAwRZ6ATtAkRT64i12/Nm2MRoldN7kDn+6xwGcXnOqClpfvIuauuL7736YVFMWZ4r+cQu09+INOTLRkPu8ZcIzH8APqTrJ/GM5OuORSwS3CNpnRFKJmNviF2u5QF7bJjKYQNbPxVKF1VdgmM3BT4/rVqOVqVLsafgDMrRLo5XSyXcHgoM8Dmy9YFxj04O8Nfx5jDrF8Wjl3o2qaqZTOXfkRFJTBjJBDxRqLA9eMJ3z2BnIYytk9LZ3x0j9lkoerw+kd3wG4yXvOHYv9vHibJQlblDKMAhhL4UTCYemY5CkHtOLdt8IrQQ1zVIuUIp0bGb8lSYdjSHj+I9djHM9JatKLx9QLXZJ1rMiXxcB/63CFxpDMjmrZ6p3gjLyVVeWtVNTkrQoqq0xbmSdnrQYNhwVnNRx0Ijir981ZPQGYOiDOSzRVZJlfLkOPv0nVdOYw2ICOGawTxNqp8fCPu8nBoPRh6ewDRMuudE8g2gChdQDaAJ9V4NkAdXVA1wBzVVjZQJ51eNrAsFWY20CadTjawKydQLIOGPMlvBIQa1v/ew6ctWOxWcRWy/SerRgsPf2LP/kW3Klopom09vXGbnAZUI8bWgw14L09TV2pLSB6NBobY7MObc+GA2vrl7gJAxXUsny5DY06GRBUBw0Ej8zACSAogKAAgq8EgilSENhuQNjuLvIfSMJBe7yyQ3o4+7jMCpSK4p7HfN0LsgLzDRDzFf3p1y73NAWARdF1NIg043o8avKW1/oNeLrjQcLtffoBCfUqJGyJaEO4BRKePqJt2EvlPV0pF5BQQMIPBwmbAEPgw97gw1YE94qgOrMTwh0hpm4XR/ezdGb/mDqwhGhTKognf5bP7YpKU5tyqg8BevnDej5A75gw5W3j88xBYw6zn5jjlWFtx63MuaLzTh5kKILzRHDegcF5Apq8lrqakhn9Su7TYX2arFy+znxo6BwUuciKfB70KC/nrUTQ3LmC5rTRWLFGLwuaQ0i/gof3gYLmtubfn3A5pDRJsbOEyxXTpWHC0yMjOkGJCUpMUGKvpMTagIsgxXqDPI9FiindAPEAVuwcBBTem4AyBstAbZ/+QH18T53868gb/E44KOPU9RAklCChxA7RUwMA4uRj+3NuH3d6/WJ+vloEo7TEPWiDVLkxce5nKSJo4yHamYUSZGjlT841x69FQiO1OcVXW6b4J496wYN2/0d2mL2Z4j8zEBLTwbLruhIhlilppkMlgogjYWJhDya6GjWOHCLegxn+oOe/YuJ52omn2r2LamATT2vfiWeOcoc48VQHvgWnp3twXjdjs97HxLPoDmLiKSaeYuI5CADwfMzDJGAr94ZFi7w+OlnwiV5ox8tqrMMR3lD0nS6WQRq6e2BUhcOV9nZKP49grE4AI0IrehVaId5HtEdoRa0PnI97wciqkC+9eR2RNWgIfGTQ2Bvy5T0iEhFf8b75pSpI2sGYl+gjYiwGQXWhbqR4yM6j7Y6hyn6h3fahvu8YKhae93in92B5MzRs0IB6ihr6GunwRu/0FryZ4M3Oj1IFb/b2EZu70VygideiibNsQE4vHUUR2ZQy5DzhruQ7LthRIaj2+hW16j3gICtxZ5db1Y4QazqUD3R2qy+gS5+gi9jwLKCLgC5Dgi4v+1DIyHFoDFYhf2PB4etyUVrIHiSL0QmLDlo/yl9VV1semhhIRTeN5aH650j7sjKUAg0aXT/QbF0SPb1axFKceHGmxR+k9HRzbeGAB4qGjo0fxPKPWP4Ryz+vfePc0w7zDCtAAu6chKnp3mXzkZma4pOUzzM11hEBxhtv1B32Th2lp1t1+voN1LchT04e9y3IE0GeCPLkjT9vUHxT9YSB0l/IKnTmb83KKN27vAQrMxhWRkNqhZXpTUiu0qNtaWLWPvhZu3D3787dP/Pq/9T51tzjGTiYt5m3HGsWNkhCRj1d6EzxCYCL3QcAdp8DeMUnALYsz9CIHf39EzvqsKNi1HcVFVMYnCB2BLEjkN4HR3o9I3amKl/d444sPvyt/+pefIveiXDEpvRebUrX5ZGqGi/blK4YBkIfaVM6GP75WC+9AEu9Y730QWPQY6O2wcQiUapptmoRydFsW9JkwCm2pSMJcKRJHWSp1DwyPO9BLFLfOb/uF+GXnbiIzek96LqLWH7hNCHO/eHAKytlD3ZJ6cReJ1jreuIFNcNf68ravD/vn0EY98PtF+/LG6jbP/br/gbj9odIBYjFzPcNbBqOUoCbNwM3vNUYS0pZP3Hr+cp7Kwj/Dw== \ No newline at end of file