Skip to content

Latest commit

 

History

History
280 lines (192 loc) · 18 KB

10_index.md

File metadata and controls

280 lines (192 loc) · 18 KB
Raw

Documentation - Executive summary

Quick Setup

  1. Prerequisites - to setup AIFactory - Estimated setup time: 1-2h
  2. End-2-End SETUP tutorial - AIFactory + 1 ESMLProject - Estimated setup time: 4-8h

Governance related - relevant for central IT, networking team (CoreTeam: 10-29)

Consumer related - relevant for developers, data scientists, data engineers (ProjectTeam: 30-39)

FAQ - Trouble shooting - relevant for all

10) AI Factory (ESML) - About the documentation

This is the main page for all documentation, with links to underlying specifics. The docs pages is sorted within a number series, by role and by component.

Rolebased: The docs pages is sorted within a number series conneted to role, and focus area.

Example: Series 10-29 is targeting the AIFactory CoreTeam, e.g. governance of the AIFactory. Series 30-39 is relevant for AIFactory ProjectTeams, e.g. the consumers of the AIFactory services & accelerators.

Doc series Role Focus Details
10-19 CoreTeam Governance Setup of AI Factory. Governance. Infrastructure, networking. Permissions
20-29 CoreTeam Usage User onboarding & AI Factory usage. DataOps for the CoreTeam's data ingestion team
30-39 ProjectTeam Usage Dashboard, Available Tools & Services, DataOps, MLOps, Access options to the private AIFactory
40-49 All FAQ Various frequently asked questions. Please look here, before contacting an ESML AIFactory mentor.

Component based: There are 4 main components of an ESML AIFactory.

All 4 can be used, or optionally cherry picked. The 1st component, Infra:AI Factory, is a pre-requsite. Below the 4 components are seen.

This table will be used in the documentation to clarify WHAT a section covers, and for WHOM/Role

Component In section Focus in section Role Doc series
1) Infra:AIFactory Y - CoreTeam 10-19
2) Datalake template Y - All 20-29,30-39
3) Templates for: DataOps, MLOps, *LLMOps Y - All 20-29,30-39
4) Accelerators: ESML SDK (Python, PySpark), RAG Chatbot, etc Y - ProjectTeam 30-39

11) Infra:AIFactory: Static documentation (CoreTeam)

Here you will see the definition of an AIFactory via lists and diagrams, and workflows how to add projects or members to an AIFactory.

All Diagrams: Architecture & Services - High-Level & Low Level Diagrams

12) Infra:AIFactory: Roles & Permissions for users (CoreTeam)

Detailed information about Roles and permission, such as Microsoft Entra ID

Service Principals (Automation & Ops purpose) & Permissions for Users, AD groups.

Permissions & Roles: Coreteam VS Project team

13) Infra:AIFactory: Flow diagrams (CoreTeam)

Flow diagram which can explains the workflows, how to utilize the complete solution(AI Factory, MLOps Accelerator) in different scenarios.

14) Infra:AIFactory: Networking: Private DNS zones, Hub/Spoke etc (CoreTeam)

HOWTO - Networking: Hybrid access, Private DNS Zones, etc

20) Infra:AIFactory - How-to: Onboarding, Roles & Permission described (CoreTeam)

Component In section Focus in section Role in section Index
1) Infra:AIFactory Y Usage & Onboard teams CoreTeam:Infra 21
2) Datalake template Y Setup CoreTeam:Infra 22
3) Templates for: DataOps, MLOps, *LLMOps Y DataOps CoreTeam: DataIngestion 23
4) Accelerators: ESML SDK (Python, PySpark), RAG Chatbot, etc N - - -

User onboarding, permissions and usage howto. There are 2 roles in the AIFactory, here the different permissions roles have is explained.

  • AIFactory CoreTeam:
  • AIFactory ProjectTeam:

21) Infra:AIFactory- How-to: Onboarding of CoreTeam users and ProjectMembers via PIPELINES (CoreTeam)

See Roles and permissions. See Flow Diagram - Add AIFactory project, Add users

Option A) GitHub Actions workflow

Option B) Azure Devops workflow

22) Datalake template- How-to: Setup Datalake & Onboard ProjectTeam via PIPELINES

Here you can find HOWTO guides for the ESML CoreTeam, how to setup the Datalake structure, and how to provide a ProjectTeam access to their datalake projectfolder, by running a pipeline (ADO, GHA)

23) Templates: DataOps - How-to: Setup DataOps via PIPELINE templates

Here you can find HOWTO guides for the ESML CoreTeam, its Dataingestion team within the CoreTeam.

24) End-2-End setup tutorial: AIFactory (4-8 hours) - How-to

Prerequisites - to setup AIFactory

Here is an End-2-End setup tutorial, with an estimated ~4-8 hour setup time.

  • Estimated setup time is 3-7 hours, to have the full AIFactory automation configured, and create "AIFactory Common DEV" + the 1st AIFactory project created (type:ESML)
  • Estimated setup time 1 hour: Configure AIFactory Common DEV + the 1st ESMLProject (type: ESML)

After the setup, you can simply click on a pipeline to provision 1 or 250 AIFactory project architectures, of type ESML or ESGenAI.

30) Usage: Dashboard, Available Tools & Services, DataOps, MLOps, Access options to AIFactory (ProjectTeam)

Here you can find HOWTO guides for a ESML ProjectTeam, including its DataOps, MLOps, supported use cases, accelerated use cases. Also how to get access to the private AIFactory.

Component In section Focus in section Role in section Index
1) Infra:AIFactory Y Usage & Get Access, Dashboards, Services ProjectTeam 31,32
2) Datalake template Y Usage & Get Access ProjectTeam 34
3) Templates for: DataOps, MLOps, *LLMOps Y DataOps ProjectTeam 36,37
4) Accelerators: ESML SDK (Python, PySpark), RAG Chatbot, etc Y Setup SDK & Templates ProjectTeam 33,35

31) How-to guide: Get access to the AIFactory:RBAC & Networking (ProjectTeam)

  1. Talk to your AIFactory CoreTeam. Ask them to onboard you, as per section 20)

  2. Depending on your AIFactory setup, you may need to take additional actions to get network access.

32) Overview: Dashboards, Services & Acceleration in AIFactory (ProjectTeam, CoreTeam)

32.1 Dashboards in the AI Factory

There are multiple ESML AIFactory dashboards available, that the coreteam shared to you.

How-to guide: IMPORT AIFacotory dashboards or select them, clone them, to customize them further

32.2 Azure services available in the AI Factory

There are Azure services packaged both for DataOps, MLOps, and Generative AI.

SERVICES LIST: Overview of the services, as a list, with naming convetions

SERVICES ARCHITECTIRE - architectural diagrams

32.3 Supported & Accelerated use cases (AI Factory - ProjectTeam)

Here you can find information about supported use cases, and accelerated use cases.

HOWTO - Supported use cases & Accelerated use cases

33) Setup: Install AzureML SDK v1+v2 and ESML accelerator library

Here you see HOWTO install AzureML SDK v1+v2 and ESML accelerator library, locally, or at an Azure Machine Learning Compute Instance.

HOWTO - install AzureML SDK v1+v2 and ESML accelerator library

34) Setup: Datalake - Project folder: How-to onboard your own data (ProjectTeam - R&D purpose)

  1. Prerequisite: Datalake access HOW-TO: Setup Datalake & Onboard ProjectTeam permissions (CoreTeam)
  2. HOWTO - Quickstart: Onboard own data, to the datalake

35) Setup: ESML SDK accelerated Notebook templates

The ESML AIFactory comes with Notebook templates, generic notebooks (not examples). See below: How to clone them, and configure them, to accelerate your projects. It also comes with DEMO data, per project team.

Pre-requisite: Get Datalake access - See section 34

HOWTO - Quickstart: Copy AI Factory notebook templates

HOWTO - Quickstart: Configure templates, and data settings - train model with own data

36) How-to guide: DataOps

HOWTO - Quickstart: DataOps with Azure Datafactory ESML templates

37) How-to guide: MLOps

  1. START HERE - WHAT is accelerated & WHAT is vanilla Azure ML, Azure Databricks?

  2. HOWTO - Quickstart: Accelerate MLOps - with ESML boost on Azure Machine Learning and Databricks

  3. HOWTO - Quickstart: MLOps with Azure Machine Learning and Databricks

38) How-to guide: LLMOps & RAG Chat Agent

!WIP! - HOWTO - Quickstart: LLMOps & RAG Chat Agent

39) End-2-End setup: ESML Project, ESGenAI Project

ESML Project: Here is an end-2-end setup tutorial, for DataOps and MLOps

ESGenAI Project: Here is an end-2-end setup tutorial, for ESML GenAI project

  • Estimated time to setup is 1-2 hours
  • ESGenAI project configured with "on your data" with Azure AI Search, Promptflow with multiple indexed.
  • Re-indexed on "code changed" (CI/CD trigger on promptflow) and redeployed endpoint
  • Deployed as an online endpoint, with possibility to scale with multiple backends (round-robin)
  • Batch Evaluation & Live Monitoring
  • !WIP! HOWTO - Quickstart: MLOps with Azure Machine Learning and Databricks

[TODO - Link]

40) FAQ

Here you can browse some Q's, and jump into relevant section. We tried to group the FAQ's into different roles & services

41) FAQ - Core team & AFactory infra

Example quetions:

  • Q: How-to clone repo with submodule to local computer? the folder azure-enterprise-scale-ml is empty?
  • Q: Why can't it find the path in my submodule?
  • Q: DataOps: How to work with Azure DataFactory, and branching?
  • Q: MLOps: Azure Devops and GIT Branching stragey - DEV, TEST, PROD, many models branches?

FAQ - FAQ-1

42) FAQ - Data scientist & Azure ML pipelines

Example quetions:

  • Q: How-to clone repo with submodule to local computer? the folder azure-enterprise-scale-ml is empty?
  • Q: Why can't it find the path in my submodule?
  • Q: DataOps: How to work with Azure DataFactory, and branching?
  • Q: MLOps: Azure Devops and GIT Branching stragey - DEV, TEST, PROD, many models branches?

FAQ - FAQ-1

43) FAQ - Data engineering & Azure ML pipelines

Example questions:

  • Q: Why does Azure Datafactory not trigger DataMesh copy to MASTER from PROJECt, when project pipeline is finished?
  • Q: How to setup eventdriven

FAQ - FAQ-43