Skip to content

Commit

Permalink
Merge pull request #139 from eccenca/bugfix/reviewOutdatedScreenshots…
Browse files Browse the repository at this point in the history
…-ECC-5926

Bugfix/review outdated screenshots ecc 5926
  • Loading branch information
rpietzsch authored Aug 3, 2023
2 parents 33aedff + 66390a8 commit e9a29d4
Show file tree
Hide file tree
Showing 70 changed files with 300 additions and 280 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The workflow is then called via an HTTP REST call (or via [cmemc](../cmemc-comm

This allows for solving all kinds of [☆ Automation](../) tasks when you need to process lots of small data snippets or similar.

!!! Abstract
!!! Abstract

The complete tutorial is available as a [project file](tutorial-varinput.project.zip). You can import this project

Expand All @@ -35,15 +35,15 @@ First, install all required ontologies/vocabularies which are needed for mapping

In this tutorial, we need the _Schema.org_ and the _RDFS_ vocabulary. Press the (toggle switch) button on the right to install them.

![Vocabulary catalog](pdwviw-vocab-catalog.png)
![Vocabulary catalog](pdwviw-vocab-catalog.png){ class="bordered" }

## 2 Create a new project

Second, create in the tab **DATA INTEGRATION** a new project. Provide it with a _Title_ and _Description_.

The project will include everything you need to build a workflow for extracting Feed XML data, transforming it into RDF, and loading it into a Knowledge Graph.

![Create new project dialog](pdwviw-create-new-project.png)
![Create new project dialog](pdwviw-create-new-project.png){ class="bordered" }

## 3 Create an (example) feed dataset and target graph dataset

Expand All @@ -54,7 +54,7 @@ For this tutorial, you may take this file: [feed.xml](feed.xml)(1)

1. Original feed source was: `https://www.ecdc.europa.eu/en/taxonomy/term/2942/feed`

![Create XML dataset dialog](pdwviw-create-xml-dataset.png)
![Create XML dataset dialog](pdwviw-create-xml-dataset.png){ class="bordered" }

## 4 Create the feed transformation

Expand All @@ -64,14 +64,14 @@ The mappings are based on classes and properties defined by the _Schema.org_ and

In case you need help with mapping data from XML to RDF, feel free to visit your respective tutorial: [Lift data from JSON and XML sources](../../build/lift-data-from-json-and-xml-sources/).

![Feed transformation](pdwviw-feed-transformation.png)
![Feed transformation](pdwviw-feed-transformation.png){ class="bordered" }

## 5 Create the variable input and workflow

Create a new workflow in your project.
Move the **input** XML feed dataset and the **Feed Data** Graph into the workflow editor and connect them with your created **Transform feed**.

![Workflow with variable input](pdwviw-variable-input-workflow.png)
![Workflow with variable input](pdwviw-variable-input-workflow.png){ class="bordered" }

## 6 Use `cmemc` to feed data into the workflow

Expand Down Expand Up @@ -107,4 +107,4 @@ http://feeds.bbci.co.uk/news/rss.xml

In **EXPLORATION**, you can study the ingested feed data in your Knowledge Graph.

![Explore the result](pdwviw-review-knowledge-graph.png)
![Explore the result](pdwviw-review-knowledge-graph.png){ class="bordered" }
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/build/define-prefixes-namespaces/DM-query-editor.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/build/define-prefixes-namespaces/DM-turtle-editor.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
27 changes: 13 additions & 14 deletions docs/build/define-prefixes-namespaces/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,23 @@ Namespace declarations allow for the abbreviation of IRIs by using a prefixed re

For example, after defining a namespace with the values

- **prefix name** = `cohw`, and the
- **namespace IRI** = `https://data.company.org/hardware/`
- **prefix name** = `cohw`, and the
- **namespace IRI** = `https://data.company.org/hardware/`

you can use the term `cohw:test` as an abbreviation for the full IRI `https://data.company.org/hardware/test`.

This is particularly useful when you have to write source code in Turtle and SPARQL.

## Managing Namespace Declarations


### Using the Vocabulary Catalog

After installing a vocabulary from the [Vocabulary Catalog](/explore-and-author/vocabulary-catalog), the vocabulary namespace declaration is automatically added to all integration projects.

In order to get the **prefix name** and the **namespace IRI** from the vocabulary graph, the following terms from the [VANN vocabulary](https://vocab.org/vann/) need to be used on the Ontology resource.

- [vann:preferredNamespacePrefix](https://vocab.org/vann/#preferredNamespacePrefix) - to specify the **prefix name**
- [vann:preferredNamespaceUri](https://vocab.org/vann/#preferredNamespaceUri) - to specify the **namespace IRI**
- [vann:preferredNamespacePrefix](https://vocab.org/vann/#preferredNamespacePrefix) - to specify the **prefix name**
- [vann:preferredNamespaceUri](https://vocab.org/vann/#preferredNamespaceUri) - to specify the **namespace IRI**

In the Explore area, an Ontology with a correct namespace declaration looks like this:

Expand All @@ -43,7 +42,7 @@ The `vocabulary` command group of [cmemc](../../automate/cmemc-command-line-inte
Beginning with v22.2, this command has an additional option `--namespace` which you can use to set a vocabulary namespace even if the vocabulary does not include the data needed for autodiscovery:

```shell-session
$ cmemc vocabulary import my-ont.ttl --namespace myo https//example.org/my/`
cmemc vocabulary import my-ont.ttl --namespace myo https//example.org/my/`
```

### Using the Project Configuration
Expand All @@ -52,10 +51,10 @@ In addition to the used vocabulary namespace declarations, you may want to add w

Such organization use cases include:

- Namespaces per class / resource type:
- **prefix name** = `persons`, **namespace IRI** = `https://example.org/data/persons/`
- Namespaces per data owner or origin:
- **prefix name** = `sales`, **namespace IRI** = `https://example.org/data/sales/`
- Namespaces per class / resource type:
- **prefix name** = `persons`, **namespace IRI** = `https://example.org/data/persons/`
- Namespaces per data owner or origin:
- **prefix name** = `sales`, **namespace IRI** = `https://example.org/data/sales/`

Prefixes in Data Integration are defined on a project basis. When creating a new project, a list of well-know prefixes is already declared.

Expand All @@ -69,15 +68,15 @@ By using the **Edit Prefix Settings** button in this Configuration area, you wil

In this dialog, you are able to

- Delete a namespace declaration → **Delete Prefix**
- Add a new namespace declaration → **Add**
- Delete a namespace declaration → **Delete Prefix**
- Add a new namespace declaration → **Add**

## Validating Namespace Declarations

After adding namespace declarations to a project you are able to use the abbreviated IRIs in the user interface, for instance, in the mapping editor, the Turtle editor or the Query editor:

![Prefixes displayed in the mapping dialogs, e.g. property selection.](prefix-displayed-in-mapping.png)

![Query Editor](DM-query-editor.png)
![Query Editor](DM-query-editor.png){ class="bordered" }

![Turtle Editor](DM-turtle-editor.png)
![Turtle Editor](DM-turtle-editor.png){ class="bordered" }
65 changes: 33 additions & 32 deletions docs/build/lift-data-from-json-and-xml-sources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ The documentation consists of the following steps, which are described in detail

The following material is used in this tutorial:

- Sample vocabulary describing the data in the JSON and XML files: [products_vocabulary.nt](products_vocabulary.nt)
- Sample vocabulary describing the data in the JSON and XML files: [products_vocabulary.nt](products_vocabulary.nt)

![Visualization of the "Products Vocabulary".](products-vocab-xml+json.png)
![Visualization of the "Products Vocabulary".](products-vocab-xml+json.png){ class="bordered" }

- Sample JSON file: [services.json](services.json)
- Sample JSON file: [services.json](services.json)

```json
[
Expand All @@ -56,7 +56,8 @@ The following material is used in this tutorial:
]
```

- Sample XML file: [orgmap.xml](orgmap.xml)
- Sample XML file: [orgmap.xml](orgmap.xml)

```xml
<orgmap>
<dept id="73191" name="Engineering">
Expand Down Expand Up @@ -110,19 +111,19 @@ The vocabulary contains the classes and properties needed to map the source data

1. In Corporate Memory, click **Vocabularies** in the navigation under **EXPLORE** on the left side of the page.

![Menu entry EXPLORE > Vocabularies](menu-explore-vocabularies.png){width="30%"}
![Menu entry EXPLORE > Vocabularies](menu-explore-vocabularies.png){ class="bordered" width="30%" }

2. Click **Register new vocabulary** on the top right of the **Vocabulary catalog** page in Corporate Memory.

![Vocabularies Catalog](vocab-catalog-xml+json.png)
![Vocabularies Catalog](vocab-catalog-xml+json.png){ class="bordered" }

3. Define a **Name**, a **Graph URI** and a **Description** of the vocabulary. _In this example we will use:_

- Name: _**Product Vocabulary**_
- Graph URI: _**http://ld.company.org/prod-vocab/**_
- Description: _**Example vocabulary modeled to describe relations between products and services.**_
- Name: _**Product Vocabulary**_
- Graph URI: _**<http://ld.company.org/prod-vocab/>**_
- Description: _**Example vocabulary modeled to describe relations between products and services.**_

![Dialog to register a new vocabulary.](dialog-register-new-vocabulary.png){width="50%"}
![Dialog to register a new vocabulary.](dialog-register-new-vocabulary.png){ class="bordered" width="50%" }

4. Click **REGISTER**.

Expand All @@ -136,23 +137,23 @@ To add the data files, click Projects under BUILD in the navigation on the left

2. In **Create new item** window, select **JSON** and click Add.

![Dialog to create new JSON dataset](create-dataset-JSON.png)
![Dialog to create new JSON dataset](create-dataset-JSON.png){ class="bordered" }

3. Define a **Label** for the dataset and upload the [services.json](services.json) file. You can leave all the other fields at default values.

![Dialog to create new JSON dataset](dialog-create-new-json-dataset.png){width="45%"} ![Dialog to create new JSON dataset](dialog-create-new-json-dataset-2.png){width="45%"}
![Dialog to create new JSON dataset](dialog-create-new-json-dataset.png){ class="bordered" width="45%"} ![Dialog to create new JSON dataset](dialog-create-new-json-dataset-2.png){ class="bordered" width="45%"}

4. Click **Create**.

=== "XML"

1. Press the **Create** button and select XML

![Dialog to create new XML dataset](create-dataset-XML.png)
![Dialog to create new XML dataset](create-dataset-XML.png){ class="bordered" }

2. Define a **Label** for the dataset and upload the [orgmap.xml](orgmap.xml) example file. You can leave all the other fields at default values.

![Dialog to create new XML dataset](dialog-create-new-xml-dataset.png)
![Dialog to create new XML dataset](dialog-create-new-xml-dataset.png){ class="bordered" }

3. Click **Create**.

Expand All @@ -162,7 +163,7 @@ To add the data files, click Projects under BUILD in the navigation on the left

2. In **Create new item** window, select **Knowledge Graph** and click **Add**. The Create new item of type Knowledge Graph window appears.

![Dialog to create new Knowledge Graph dataset](create-dataset-KG.png)
![Dialog to create new Knowledge Graph dataset](create-dataset-KG.png){ class="bordered" }

3. Fill in the required details such as Label and Description.

Expand All @@ -173,7 +174,7 @@ To add the data files, click Projects under BUILD in the navigation on the left
- Name: _**Service Knowledge Graph**_
- Graph: _**http://ld.company.org/prod-instances/**_

![Dialog to create new Knowledge Graph dataset](create-new-kg-for-json.png)
![Dialog to create new Knowledge Graph dataset](create-new-kg-for-json.png){ class="bordered" }

=== "XML"

Expand All @@ -182,7 +183,7 @@ To add the data files, click Projects under BUILD in the navigation on the left
- Name: _**Organization Knowledge Graph**_
- Graph: _**http://ld.company.org/organization-data/**_

![Dialog to create new Knowledge Graph dataset](create-new-kg-for-xml.png)
![Dialog to create new Knowledge Graph dataset](create-new-kg-for-xml.png){ class="bordered" }

## 4 Create a Transformation

Expand All @@ -192,7 +193,7 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran

2. On the **Create New Item** window, select **Transform** and click **Add** to create a new transformation.

![Create new Transformation](create-new-tf.png)
![Create new Transformation](create-new-tf.png){ class="bordered" }

3. In the **Create new item of type Transform** window, enter the required fields.

Expand All @@ -205,7 +206,7 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran
- Select the Source Dataset: _**Services JSON**_
- Select the Output Dataset: _**Service_Knowledge_Graph**_

![Dialog to create new Transformation](create-new-tf-for-json.png)
![Dialog to create new Transformation](create-new-tf-for-json.png){ class="bordered" }

Click **Create**.

Expand All @@ -219,15 +220,15 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran
- Type: _**dept**_ (define the Source Type, which defines the XML element that should be iterated when creating resources)
- Select the Output Dataset: _**Organization_Knowledge_Graph**_

![Dialog to create new Transformation](create-new-tf-for-xml.png)
![Dialog to create new Transformation](create-new-tf-for-xml.png){ class="bordered" }

Click **Create**.

4. Expand the ![Mapping Button](button-mapping.png) menu by clicking the arrow on the right side of the page to expand the menu.

5. Click **Edit** to create a base mapping.

![Mapping header configuration.](mapping-configuration-header.png)
![Mapping header configuration.](mapping-configuration-header.png){ class="bordered" }

6. Define the **Target entity type** from the vocabulary, the **URI pattern** and a **Label** for the mapping.

Expand Down Expand Up @@ -280,13 +281,13 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran

7. Evaluate your mapping by pressing on the ![Button expand](button-expand-xml+json.png) button in the **Examples of target data** property to see at most three generated base URIs.

![Preview of mapping results](mapping-value-preview.png)
![Preview of mapping results](mapping-value-preview.png){ class="bordered" }

We have now created the Service entities in the Knowledge Graph. Next we will now add the name of our entity.

8. Click the circular blue **+** icon on the lower right and select **Add value mapping**.

![Add a mapping rule](service-mapping-add-rule.png)
![Add a mapping rule](service-mapping-add-rule.png){ class="bordered" }

=== "JSON"

Expand All @@ -298,7 +299,7 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran
- which corresponds to the following element in the json-file: _[ {"ProductManager": {  "name": "Corinna Ludwig"} ... } ...]_
- An optional Label: _**has Product Manager**_

![Configuration of a mapping rule](mapping-rule-config-json.png)
![Configuration of a mapping rule](mapping-rule-config-json.png){ class="bordered" }

Click **Save**.

Expand All @@ -312,39 +313,39 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran
- which corresponds to the `department name` attribute in the XML file
- An optional Label: _**department name**_

![Configuration of a mapping rule](mapping-rule-config-xml.png)
![Configuration of a mapping rule](mapping-rule-config-xml.png){ class="bordered" }

Click **Save**.

By clicking on the ![Button expand](button-expand-xml+json.png) button in the **Examples of target data** property, you can get a preview for 3x value mapping to be created.

=== "JSON"

![Mapping result](mapping-result-json.png)
![Mapping result](mapping-result-json.png){ class="bordered" }

=== "XML"

![Mapping result](mapping-result-xml.png)
![Mapping result](mapping-result-xml.png){ class="bordered" }

## 5 Evaluate a Transformation

Click **Transform evaluation** to evaluate the transformed entities.

![Transformation evaluation view](xml+json-transform-evaluation.png)
![Transformation evaluation view](xml+json-transform-evaluation.png){ class="bordered" }

## 6 Build the Knowledge Graph

1. Click **Transform execution**
2. Press the ![Button play](button-play-xml+json.png) button and validate the results. In this example, 9x Service entities were created in our Knowledge Graph based on the mapping.
3. You can click **Knowledge Graphs** under **EXPLORE** to (re-)view of the created Knowledge Graphs
4. Enter the following URIs in the Enter search term for JSON and XML respectively.
- JSON / Service: _**http://ld.company.org/prod-instances/**_
- XML / Department: _**http://ld.company.org/organization-data/**_
- JSON / Service: _**<http://ld.company.org/prod-instances/>**_
- XML / Department: _**<http://ld.company.org/organization-data/>**_

=== "JSON"

![Service KG](services-kg.png)
![Service KG](kg-services.png){ class="bordered" }

=== "XML"

![Organization KG](kg-organization.png)
![Organization KG](kg-organization.png){ class="bordered" }
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit e9a29d4

Please sign in to comment.