01_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "98e0570f-1073-457f-807f-809843d1322c",
   "metadata": {},
   "source": [
    "# MLOps for Research Software Engineers: 1. Introduction\n",
    "\n",
    "This is a tutorial intended for Research Software Engineers as an introduction and getting started guide to the tools and principles of developing machine learning-based research software according to the ML Ops principles.This was first delivered as a walkthrough as part of the [2022 UK RSE Research Software Engineering conference](https://rsecon2022.society-rse.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc6474f9-374b-4e38-baac-1d3ca94598fa",
   "metadata": {},
   "source": [
    "### About the Author\n",
    "\n",
    "*Stephen Haddad: Senior Scientific Software Engineer, Met Office Informatics Lab*\n",
    "\n",
    "Contact details:\n",
    "\n",
    "- twitter: [@stevehadd](https://twitter.com/stevehadd)\n",
    "- GitHub: https://github.com/stevehadd\n",
    "- [LinkedIn](https://www.linkedin.com/in/stephen-haddad-499a865/)\n",
    "\n",
    "I am a Scientific Software Engineer in the [Informatics Lab](https://www.metoffice.gov.uk/research/foundation/informatics-lab/index) at the [Met Office](https://www.metoffice.gov.uk/). I work in Infrastructure and Data Engineering Research and am interested in the tools and practices that enable scalable, reproducible, interactive machine learning workflows.\n",
    "\n",
    "Thanks to all my colleagues in the Informatics Lab for their assistance developing this material.\n",
    "\n",
    "![Met Office Logo](https://www.metoffice.gov.uk/webfiles/1661941781161/images/icons/social-icons/default_card_315.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8b96b8a-25e2-447c-a548-61bfcaac0e5a",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "To make use of this material, the following underpinning knowledge is required:\n",
    "\n",
    "- good knowledge of python and scientific python libraries\n",
    "- general good software development and software engineering principles and practices\n",
    "\n",
    "To the get the most out of this material, some knowledge or skill in the following areas would be helpful, but is not assumed or required:\n",
    "\n",
    "- general terminology and principles of machine learning\n",
    "- basic understanding of DevOps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94877eac-a6ca-4d87-be91-8c9367ad7970",
   "metadata": {},
   "source": [
    "### Key Outcomes\n",
    "\n",
    "You should expect to gain the following knowledge and skills from working through this material:\n",
    "\n",
    "- Understand what tools/practices that are new/different for MLOps\n",
    "- Understand how to translate standard RSE principles and practices into ML projects\n",
    "- Understand how software sustainability applies to ML projects\n",
    "- Be able to start using standard MLOps tools in a research project\n",
    "\n",
    "The intended outcome can be summarised as helping you getting started making machine learning projects and workflows a bit more systematic and reproducible than the \"style\" of machine learning depicted in this XKCD cartoon:\n",
    "![XKCD 1838](https://imgs.xkcd.com/comics/machine_learning_2x.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1cf903-322a-434f-9f18-24ff53d12b4e",
   "metadata": {},
   "source": [
    "## What is an RSE?\n",
    "\n",
    "Before start considering what we mean by ML Ops, lets first remind ourselves (or maybe clearly define for those who are new to the field) exactly what we mean by a Research Software Engineer or RSE. The reason for doing this is it will help us define what the tasks and responsibilities of an RSE are in a machine learning project compared to the responsibilities of the researchers themselves. A good place to start is the definition of an RSE from the RSE association site:\n",
    "\n",
    "*A Research Software Engineer works with researchers to gain an understanding of the problems they face, then develops, maintains and extends software to provide the answers*.\n",
    "\n",
    "Often the researchers themselves will do this to a certain extent as well and there exists a continuum from researcher to software engineer along which an RSE falls, so some questions are usually helpful in determining if you are an RSE, such as:\n",
    "\n",
    "- employed to develop software for research\n",
    "- spend more time developing software than doing research\n",
    "- you develop software that underpins the published research of others, rather than mainly publishing papers yourself\n",
    "\n",
    "The above definition doesn't capture though (in my opinion) the key areas where RSEs add value and where a project is more efficient by having an RSE supporting it even if the researchers themselves are all very capable programmers and comfortable with IT infrastructure. The key purposes/roles of of RSEs on a project and where they add value are to:\n",
    "\n",
    "- Provide solid technical foundation for research work (scalable, reliable, reusable)\n",
    "- Ensure research software assets are FAIR\n",
    "- Provide the technical underpinnings for reproducible research"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd481f91-00e6-4335-a85f-2a5f89d0ffaf",
   "metadata": {},
   "source": [
    "### What are the key principles for RSEs?\n",
    "\n",
    "A key part of the work of an RSE is applying good software engineering principle and practices to the development, maintenance and deployment of research software. As you have seen from the above description of an RSE, there are certain key principles about\n",
    "\n",
    "- *reproducibility* -  Ensure that research results are reproducible, by the original researchers and others building on their work.\n",
    "- *code quality* - Ensure good coding practices are followed so that the software is easy to adapt/extend to meet new research requirements.\n",
    "- *code performance / optmisation* - Ensure the implementation is optimised to allow it to run on large problems and datasets without wasting computational resources.\n",
    "- *reliability / robustness*. - \"harden\" code created by researchers to ensure it works reliably according to the the requirements and is available for research when required.\n",
    "- *automation* - Reduce the burden of deploying and maintaining research software and associated infrastructure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c45a1f89-dd51-4f91-9e54-4484afd7e778",
   "metadata": {},
   "source": [
    "### What the key tasks and responsibilities of RSEs?\n",
    "\n",
    "- Working with researchers to translate a research problem / question into software requirements\n",
    "- Managing research software assets like code and data\n",
    "- Applying good software engineering principles to research software\n",
    "- Ensuring research software is tested and documented\n",
    "- Deploying research software on scalable, reliable infrastructure\n",
    "\n",
    "The aim of this workshop is to help build on these existing principles and practices by extending them for the additional requirements of machine learning. This workshop will hopefully  provide RSEs a “recipe book” and \"toolset\" (to mix my metaphors) for supporting good practice in research projects that use machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f422529-4416-41d0-9e3d-1840d27fb483",
   "metadata": {
    "tags": []
   },
   "source": [
    "## What is MLOps\n",
    "\n",
    "Now that we've defined part of the the title of this workshop (the target audience i.e. RSEs), now lets better define the core subject matter of MLOps. What is MLOps and why should an RSE care about it?  **MLOps** stands for **Machine Learning Operations** , and is derived from the term DevOps, or Developer / Operations. So what is DevOps then?\n",
    "\n",
    "§"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dfbb93c-0524-48af-ad77-cd3cb1101592",
   "metadata": {},
   "source": [
    "### What is DevOps\n",
    "\n",
    "According to AWS: *DevOps is the combination of cultural philosophies, practices, and tools that increases an organisation’s ability to deliver applications and services at high velocity*. The idea is to bring together good software practices to be able to produce good quality software more quickly. In the context of research this about applying those practices to software and infrastructure that enable delivery of high quality, timely research.\n",
    "\n",
    "What are the key principles in DevOps? (https://about.gitlab.com/topics/devops/)\n",
    "\n",
    "- Automation of the software development lifecycle\n",
    "- Collaboration and communication\n",
    "- Continuous improvement and minimization of waste\n",
    "- Hyperfocus on user needs with short feedback loops\n",
    "\n",
    "![DevOps Lifecycle](https://about.gitlab.com/nuxt-images/topics/lifecycle-2.png)\n",
    "\n",
    "Further reading:\n",
    "\n",
    "- https://aws.amazon.com/devops/what-is-devops/\n",
    "- https://about.gitlab.com/topics/devops/\n",
    "- https://www.pagerduty.com/resources/learn/essential-devops-roles/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b490c79d-514f-4499-b3ce-8b282f1892d1",
   "metadata": {},
   "source": [
    "### What is MLOps\n",
    "\n",
    "MLOps aim to apply a similar philosophy to software that prepares, trains and deploys machine learning models. So ML Ops is about philosophies, practices and tools that allow organisations to create and deploy machine learning based systems quickly, efficiently and robustly. For RSEs this is specifically to be able to deliver research that relies on machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54a4c73b-b0ca-48e4-b84e-558034329e05",
   "metadata": {},
   "source": [
    "### What are the tasks of an MLOps Engineer?\n",
    "\n",
    "https://www.bestdevops.com/what-are-roles-and-responsibilities-of-mlops-engineers/\n",
    "\n",
    "An MLOps engineer is focused then on all the tasks and responsibilities around the technical infrastructure that underpin machine learning system use and promotion of good development practices within machine learning projects to ensure timely, efficient and reliable delivery of the required outcomes and deliverables. These tasks include:\n",
    "\n",
    "1. Deployment of machine learning systems, with tasks including:\n",
    "   - Optimization of model hyperparameters\n",
    "   - Automated model training and retraining pipelines\n",
    "   - Integrating machine learning models into other software systems\n",
    "   - Version control and governance for models\n",
    "   - Curation of data\n",
    "   - Curation of machine learning models\n",
    "1. To measure and improve services, create and use benchmarks, metrics, and monitoring.\n",
    "   - Evaluation of models and ongoing model monitoring\n",
    "   - Explainability/Interpretability of ML results\n",
    "1. Providing best practises and running proof-of-concepts for automated and efficient model operations on a large scale.\n",
    "1. Creating and maintaining scalable MLOps frameworks to support project-specific models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2967deb2-089b-4a14-9421-99ae43e2d690",
   "metadata": {},
   "source": [
    "### What are the required skills?\n",
    "\n",
    "- Good programming skills and knowledge.\n",
    "- Good understanding of general machine learning terms, practices and tools.\n",
    "- Understanding of the challenges and opportunities of working with big data and skills in use of data structures, data modelling, and data management systems.\n",
    "- Ability to apply quality assurance practices to code and data-framework/environments led by QA.\n",
    "- Understand the tools in the pipeline that serve different purposes, such as Continuous Integration servers, Configuration management, Deployment automation, Containers, Infrastructure Orchestration, Monitoring and Analytics, Testing, Cloud Quality tools, and network protocols.\n",
    "- Ability to automate any and all parts of a development pipeline.\n",
    "- Ability to create test infrastructure to feed into ML task like model validation, model training in addition to traditional code tests like unit and integration testing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6e741c5-683a-497c-b102-34057cde94ce",
   "metadata": {},
   "source": [
    "### RSE Perspective\n",
    "\n",
    "Why might an RSE care about adopting MLOps or DevOps practices? Many RSEs are already doing this, even if they do not realise it, as often with research software the same person or small group of people is/are responsible for both developing and supporting a software package or library. Applying principles like automation and continuous improvement can reduce the burden of supporting the software and make it easier to respond to requests from researchers for new features, extensions and improvements.\n",
    "\n",
    "In particular, machine learning is still relatively new to many researchers, and so good practices for using machine learning in research projects are not widely known and understood. Often these practices are no different to those for other research software, but the tendency to see machine learning as something new and exotic and somehow fundamentally different (phrases like the AI revolution or the fourth industrial revolution are often unhelpful and counter-productive in such a context), lead to even existing well-known practices not being used because it thought they are irrelevant or obsolete.\n",
    "\n",
    "So there is both a substantial challenge and an exciting development as more and more research projects incorporate machine learning as one of the key research tools gives opportunity for RSEs to make research that depends on ML both better and easier through the use of MLOps principles and practices applied to research software."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b74fbb6-c2e4-46b3-bb72-5f41f0d54771",
   "metadata": {},
   "source": [
    "## MLOps for RSEs - Turning principles into practice(s)\n",
    "\n",
    "I doubt I have said anything so far that an enthusiastic, diligent RSE would object to. But that RSE may be asking, how do I turn all these good ideas into something that has an effect on my day to day work? Now we'll take a look at the overall structure of a machine learning based project and what parts of it an RSE can expect to get involved with."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8f7575-3d28-45d0-bb52-d8d5732607df",
   "metadata": {},
   "source": [
    "### Machine Learning Pipeline\n",
    "\n",
    "We'll start by looking at the overall parts of a machine learning pipeline that one may build for a research project. Below is a list (more or less in the order of execution in a data pipeline), of the minimum generic components required for a well executed pipeline.\n",
    "\n",
    "- *Data Loading & Cleaning* - Start by loading the data, and filtering out any data considered to be unsuitable training and evaluation of machine learning algorithms. Selection of appropriate data is an important way in which domain expertise in vital in getting good results.\n",
    "- *Feature Engineering* - The first step is to prepare the data for presenting to the algorithm. Different ways of presenting the data will emphasise different features, and choosing the right features is important for getting good results. Knowledge of what features represent based on domain knowledge is again very important.\n",
    "- *Train/test Split* - Before we train the algorithm, we need to split into train and test sets. This is to ensure our algorithm doesn't overfit, learning irrelevant details that are not representative of the whole space of possible data, but rather that it generalises well.\n",
    "- *Data Preparation* - The machine learning algorithm only sees numbers as numbers, with no inherent understanding of meaning or context. We need to ensure different features are scaled to be comparable, otherwise big numbers will be treated as more important by the algorithm, irrespective of what those numbers mean. Value are typically scaled to a range of \\[0,1\\] or, assuming a Gaussian distribution, to have mean=0 and std_dev=1.\n",
    "- *Algorithm Setup* - Here we select the particular algorithm e,.g. neural network, k-means clustering, and specify the hyperparameters. It is important to distinguish between parameters and hyperparameters.\n",
    "  - Parameters are the values that are calculated by the training process.\n",
    "  - Hyperparameters are values specified in algorithm setup, which are not altered by training. These need to be fine-tuned using an additional outer training loop called hyperparameter tuning.\n",
    "- *Algorithm Training* - Execute the algorithm to calculate the best parameters for the chosen ML algorithm to fit the supplied training data\n",
    "- *Inference* - Once we have an algorithm, we use it to produce predictions, for both the train and test sets.\n",
    "- *Evaluation* - We then compare the predictions of the trained algorithms to expected results. For supervised learning, this will be supplied target values. For unsupervised learning, we will explore the results and their usefulness much like in exploratory data analysis.\n",
    "- *Interpretability & Explainability* - Explaining how the model produced a certain prediction (looking inside the blackbox) and interpreting what the prediction means in terms of relation to the real world problem.\n",
    "- *Model Storage* - Model training can be an expensive process that we don't want to perform too often. Once we have a model that performs well we save its state so it can be reloaded and used subsequently for inference on later problems."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53216cfa-7e5a-48f1-bd73-c0aecdd302b9",
   "metadata": {},
   "source": [
    "### ML Lifecycle from an RSE perspective\n",
    "\n",
    "![Uber Michaelangelo's ML lifecycle](https://1fykyq3mdn5r21tpna3wkdyi-wpengine.netdna-ssl.com/wp-content/uploads/2018/11/image6.png)\n",
    "Next to consider is where and how an RSE could or should get involved in the development and deployment of such a workflow. In research projects, projects generally occupy the Prototype phase of the ML Life cycle diagram, with the other parts generally happening when a project is taken on by another organisation to be commercialised or generally turned into a product of some part. So in this tutorial we are focusing on the elements of the pipeline in the prototype phase. The focus of RSE involvement in a project can generally be divided into three phases, each incorporating multiple elements of the ML workflow described above. This will form the structure for the remainder of this tutorial:\n",
    "\n",
    "- *Data Preparation and Exploration*  - Finding and aggregating data into a coherent dataset and transforming it into an analysis ready dataset for researchers to use as input to a machine learning pipeline.\n",
    "- *Model Development* Set up the tools and infrastructure required to train large machine learning models and keep track of the input and outputs.\n",
    "- *Model Evaluation (\"Evaluate models\")* - Evaluate the performance of the trained models in terms of accuracy metrics, explainability/interpretability metrics and enable sharing of models and predictions.\n",
    "\n",
    "In many ways developing software for a machine learning project is no different than for any other sort of software project, but there are some additional or different challenges and also some different tools to supports tasks that are unique to a machine learning pipeline. In order to apply the principles of good software development that are typically applied by RSEs to ML projects RSEs will need to know about what ML software development and support has in common with other sorts of projects and what is different, and what tools are needed to support the different aspects. Many of these tools are about promoting these good practices in research ML projects, where researchers may not not be familiar with those good practices or have the technical knowledge to set up and use the infrastructure and tools to implement those practices. These notebooks will demonstrate some of those tools that promote good practices leading to reproducible, scalable research software."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22bfda0-278e-4f2e-8b56-7dddc1aa42d3",
   "metadata": {},
   "source": [
    "## References:\n",
    "\n",
    "- ML Ops Wikipedia https://en.wikipedia.org/wiki/MLOps\n",
    "- Databricks MLOps https://databricks.com/glossary/mlops\n",
    "- Who is a RSE https://ukrse.github.io/who.html\n",
    "- What is Research Software Engineering https://software-carpentry.org/blog/2015/06/what-is-a-research-software-engineer.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714b4f13-3fd3-449c-a495-fb7a0141d9ee",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "- ML Engineering - https://databricks.com/wp-content/uploads/2021/09/ML-Engineering-Ebook-Final.pdf\n",
    "- Reproducible Machine L:Earning https://dev.to/azure/10-top-tips-for-reproducible-machine-learning-36g0\n",
    "- https://dagshub.com/blog/solve-your-mlops-problems-with-an-open-source-data-science-stack/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".conda-ukrse2022_mlops_data_prep Python (Conda)",
   "language": "python",
   "name": "conda-env-.conda-ukrse2022_mlops_data_prep-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}