-2. Set Environment variables.
- Library uses DVC for artifact versioning and git for code versioning. It requires a set of environment variables to operate DVC and git. The list of environment variables needed can be found in example-get-started/sample_env.
- Copy sample_env from example-get-started directory to local directory. - Modify sample_env and run 'source sample_env' to set the environment variables.
-3. Copy initialize.sh from example-get-started folder to your local directory.
- Run command - sh initialize.sh
- Before running the script, please ensure that required environment variables are set.
- This configures DVC and git with the provided variables in Step 1.
- - -## Logging metadata with CMF -#### Import of the library to record the metadata -```python -from cmflib import cmf - -``` - -### Create the metadata writer -The metadata writer is responsible to manage the backend to record the metadata. -It also creates a pipeline abstraction, which helps to group individual stages and execution. -```python -cmf = cmf.Cmf(filename="mlmd", - pipeline_name="Test-env") - -``` -### Create the stage in pipeline. -An ML pipeline can have multiple stages. This context abstraction tracks the stage and its metadata. -A dictionary can be passed to hold the user given metadata. The custom properties is an optional argument - ```python -context = cmf.create_context(pipeline_stage="Prepare", - custom_properties={"user-metadata1":"metadata_value"}) -``` - -#### Create the execution -A stage in ML pipeline can have multiple executions. Every run is marked as an execution. -This API helps to track the metadata associated with the execution -```python -execution = cmf.create_execution(execution_type="Prepare-1", custom_properties = {"user-metadata1":"metadata_value"}) -``` -#### Log artifacts -An Execution could have multiple artifacts associated with it as Input or Output. The path of the artifact provided should be path from root of the repo. -The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary -```python -cmf.log_dataset(input, "input", custom_properties={"user-metadata1":"metadata_value"}) -cmf.log_dataset(output_train, "output", custom_properties={"user-metadata1":"metadata_value"}) -cmf.log_dataset(output_test, "output", custom_properties={"user-metadata1":"metadata_value"}) -``` -#### Log model -A model developed as part of training step or used in a evaluation or inference step can be logged. It can be input or output -The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary -```python -cmf.log_model(path="model.pkl", event="output", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) -cmf.log_model(path="model.pkl", event="input", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) -``` -#### Log metrics -Metrics of each step can be logged -The metadata associated with the artifact could be logged as argument which takes in a dictionary -```python -#Can be called at every epoch or every step in the training. This is logged to a parquet file and commited at the commit stage. -while True: #Inside training loop - cmf.log_metric("training_metrics", {"loss":loss}) -cmf.commit_metrics("training_metrics") -``` -#### Log Stage metrics -Metrics for each stage. -```python -cmf.log_execution_metrics("metrics", {"avg_prec":avg_prec, "roc_auc":roc_auc}) -``` -#### Creating the dataslices -This helps to track a subset of the data. For eg- Accuracy of the model for a slice of data(gender, ethnicity etc) -```python -dataslice = cmf.create_dataslice("slice-a") -for i in range(1,20,1): - j = random.randrange(100) - dataslice.add_data("data/raw_data/"+str(j)+".xml") -dataslice.commit() -``` -### To use Graph Layer -CMF library has an optional Graph layer which stores the relationships in a Graph Database(NEO4J) -To use the graph layer, the "graph" parameter in the library init call should be set to true. This is set as false by default. -The library reads the configuration parameters of the Database from the environment variables. -The variables "NEO4J_URI", "NEO4J_USER_NAME", "NEO4J_PASSWD" should be either set as environment variables. - -``` - export NEO4J_URI="bolt://10.93.244.219:7687" - export NEO4J_USER_NAME=neo4j - export NEO4J_PASSWD=neo4j +# cmf +## Common Metadata Framework +[Getting Started](docs/README.md)
+[Detailed documentation of the API's](docs/API.md)
+ +Interactions in data pipelines can be complex. The Different stages in the pipeline, (which may not be next to each other) may have to interact to produce or transform artifacts. As the artifacts navigates and undergo transformations through this pipeline, it can take a complicated path, which might also involve bidirectional movement across these stages. Also there could be dependencies between the multiple stages, where the metrics produced by a stage could influence the metrics at a subsequent stage. It is important to track the metadata across a pipeline to provide features like, lineage tracking, provenance and reproducibility. + +The tracking of metadata through these complex pipelines have multiple challenges, some of them being, + +Each stage in the pipeline could be executed in a different datacenter or an edge site having intermittent connection to the core datacenter. + +Each stage in the pipeline could be possibly managed by different teams. + +The artifacts (input or output) needs to be uniquely identified across different sites and across multiple pipelines. + + +Common metadata framework (CMF) addresses the problems associated with tracking of pipeline metadata from distributed sites and tracks code, data and metadata together for end-to-end traceability. + +The framework automatically tracks the code version as one of the metadata for an execution. Additionally the data artifacts are also versioned automatically using a data versioning framework (like DVC) and the metadata regarding the data version is stored along with the code. The framework stores the Git commit id of the metadata file associated with the artifact and content hash of the artifact as metadata. The framework provides API’s to track the hyper parameters and other metadata of pipelines. Therefore from the metadata stored, users can zero in on the hyper parameters, code version and the artifact version used for the experiment. + +Identifying the artifacts by content hash allows the framework, to uniquely identify an artifact anywhere in the distributed sites. This enables the metadata from the distributed sites to be precisely merged to a central repository, thereby providing a single global metadata from the distributed sites. + +On this backbone, we build the Git like experience for metadata, enabling users to push their local metadata to the remote repository, where it is merged to create the global metadata and pull metadata from the global metadata to the local, to create a local view, which would contain only the metadata of interest. + +The framework can be used to track various types of pipelines such as data pipelines or AI pipelines. +
+ 
+
+ 
+
+[Detailed documentation of the API's](docs/API.md) + +# Getting Started +### Install the library + +#### Creating the wheel file + +``` +python setup.py bdist_wheel +cd dist +pip install
+2. Set Environment variables.
+ Library uses DVC for artifact versioning and git for code versioning. It requires a set of environment variables to operate DVC and git. The list of environment variables needed can be found in example-get-started/sample_env.
+ Copy sample_env from example-get-started directory to local directory. + Modify sample_env and run 'source sample_env' to set the environment variables.
+3. Copy initialize.sh from example-get-started folder to your local directory.
+ Run command - sh initialize.sh
+ Before running the script, please ensure that required environment variables are set.
+ This configures DVC and git with the provided variables in Step 1.
+ + +## Logging metadata with CMF +#### Import of the library to record the metadata +```python +from cmflib import cmf + +``` + +### Create the metadata writer +The metadata writer is responsible to manage the backend to record the metadata. +It also creates a pipeline abstraction, which helps to group individual stages and execution. +```python +cmf = cmf.Cmf(filename="mlmd", + pipeline_name="Test-env") + +``` +### Create the stage in pipeline. +An ML pipeline can have multiple stages. This context abstraction tracks the stage and its metadata. +A dictionary can be passed to hold the user given metadata. The custom properties is an optional argument + ```python +context = cmf.create_context(pipeline_stage="Prepare", + custom_properties={"user-metadata1":"metadata_value"}) +``` + +#### Create the execution +A stage in ML pipeline can have multiple executions. Every run is marked as an execution. +This API helps to track the metadata associated with the execution +```python +execution = cmf.create_execution(execution_type="Prepare-1", custom_properties = {"user-metadata1":"metadata_value"}) +``` +#### Log artifacts +An Execution could have multiple artifacts associated with it as Input or Output. The path of the artifact provided should be path from root of the repo. +The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary +```python +cmf.log_dataset(input, "input", custom_properties={"user-metadata1":"metadata_value"}) +cmf.log_dataset(output_train, "output", custom_properties={"user-metadata1":"metadata_value"}) +cmf.log_dataset(output_test, "output", custom_properties={"user-metadata1":"metadata_value"}) +``` +#### Log model +A model developed as part of training step or used in a evaluation or inference step can be logged. It can be input or output +The metadata associated with the artifact could be logged as an optional argument which takes in a dictionary +```python +cmf.log_model(path="model.pkl", event="output", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) +cmf.log_model(path="model.pkl", event="input", model_framework="SKlearn", model_type="RandomForestClassifier", model_name="RandomForestClassifier:default" ) +``` +#### Log metrics +Metrics of each step can be logged +The metadata associated with the artifact could be logged as argument which takes in a dictionary +```python +#Can be called at every epoch or every step in the training. This is logged to a parquet file and commited at the commit stage. +while True: #Inside training loop + cmf.log_metric("training_metrics", {"loss":loss}) +cmf.commit_metrics("training_metrics") +``` +#### Log Stage metrics +Metrics for each stage. +```python +cmf.log_execution_metrics("metrics", {"avg_prec":avg_prec, "roc_auc":roc_auc}) +``` +#### Creating the dataslices +This helps to track a subset of the data. For eg- Accuracy of the model for a slice of data(gender, ethnicity etc) +```python +dataslice = cmf.create_dataslice("slice-a") +for i in range(1,20,1): + j = random.randrange(100) + dataslice.add_data("data/raw_data/"+str(j)+".xml") +dataslice.commit() +``` +### To use Graph Layer +CMF library has an optional Graph layer which stores the relationships in a Graph Database(NEO4J) +To use the graph layer, the "graph" parameter in the library init call should be set to true. This is set as false by default. +The library reads the configuration parameters of the Database from the environment variables. +The variables "NEO4J_URI", "NEO4J_USER_NAME", "NEO4J_PASSWD" should be either set as environment variables. + +``` + export NEO4J_URI="bolt://10.93.244.219:7687" + export NEO4J_USER_NAME=neo4j + export NEO4J_PASSWD=neo4j + +''' +Create the metadata writer +The metadata writer is responsible to manage the backend to record the metadata. +It also creates a pipeline abstraction, which helps to group individual stages and execution. +''' +cmf = cmf.Cmf(filename="mlmd", + pipeline_name="Test-env", graph=True) + +``` + +### Use a Jupyterlab environment with CMF pre-installed +- CMF is preinstalled in a JupyterLab Notebook Environment. +- Accessible at http://[HOST.IP.AD.DR]:8888 (default token: `docker`) +- Within the Jupyterlab environment, a startup script switches context to `$USER:$GROUP` as specified in `.env` +- `example-get-started` from this repo is bind mounted into `/home/jovyan/example-get-started` +- Update `docker-compose.yml` as needed. For example to bind mount another volume on the host: `/lustre/data/dataspaces/dataspaces_testbed/:/home/jovyan/work` + +``` +#create .env file in current folder using env-example as a template. #These are used by docker-compose.yml +docker-compose up --build -d +#To Shutdown/Remove (Remove Volumes as well) +docker-compose down -v +``` diff --git a/presentations/LFAI_v2.pdf b/presentations/LFAI_v2.pdf new file mode 100644 index 00000000..65c61fcb Binary files /dev/null and b/presentations/LFAI_v2.pdf differ diff --git a/presentations/LFAI_v2.pptx b/presentations/LFAI_v2.pptx new file mode 100644 index 00000000..0337734f Binary files /dev/null and b/presentations/LFAI_v2.pptx differ