From 64654f36ff0aef81e76252e47b620662140bacf4 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Tue, 9 May 2023 16:27:46 +0300 Subject: [PATCH 01/17] wip: update cli --- docs/cli.md | 171 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 136 insertions(+), 35 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 612edca03cb..7407eb2fb2a 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -79,21 +79,62 @@ openfn test The word `openfn` will invoke the CLI. The word `test` will invoke the test command. -You should see some output like this: +
+ You should see some output like this: ```sh [CLI] ℹ Versions: - ▸ node.js 18.12.1 - ▸ cli 0.0.29 - ▸ runtime 0.0.19 - ▸ compiler 0.0.25 + ▸ node.js 18.12.1 + ▸ cli 0.0.39 + ▸ runtime 0.0.24 + ▸ compiler 0.0.32 [CLI] ℹ Running test job... -[CLI] ✔ Compiled job -[JOB] ℹ Calculating the answer to life, the universe, and everything... -[R/T] ✔ Operation 1 complete in 1ms +[CLI] ℹ Workflow object: +[CLI] ℹ { + "start": "start", + "jobs": [ + { + "id": "start", + "data": { + "defaultAnswer": 42 + }, + "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", + "next": { + "calculate": "!state.error" + } + }, + { + "id": "calculate", + "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", + "next": { + "result": true + } + }, + { + "id": "result", + "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" + } + ] +} + +[CLI] ✔ Compilation complete +[R/T] ♦ Starting job start +[JOB] ℹ Starting computer... +[R/T] ℹ Operation 1 complete in 0ms +[R/T] ✔ Completed job start in 1ms +[R/T] ♦ Starting job calculate +[JOB] ℹ Calculating to life, the universe, and everything.. +[R/T] ℹ Operation 1 complete in 0ms +[R/T] ✔ Completed job calculate in 1ms +[R/T] ♦ Starting job result +[R/T] ℹ Operation 1 complete in 0ms +[R/T] ✔ Completed job result in 0ms [CLI] ✔ Result: 42 + ``` +
+ What we've just done is executed a JavaScript expression, which we call a _job_. The output prefixed with `[JOB]` comes directly from `console.log` statements in our job code. All other output is the CLI trying to tell us what it is doing. @@ -128,6 +169,32 @@ openfn test --log debug #### Tasks: +:::info To get started with @openfn/cli + +1. Create a new folder for the repository you'll be working on by running the + following command: `mkdir devchallenge && cd devchallenge` + +2. While you can keep your job scripts anywhere, it's a good practice to store + `state.json` and `output.json` in a `tmp` folder. To do this, create a new + directory called `tmp` within your `devchallenge` folder: `mkdir tmp` + +3. Since `state.json` and `output.json` may contain sensitive configuration + information and project data, it's important to never upload them to Github. + To ensure that Github ignores these files, add the `tmp` directory to your + `.gitignore` file: `echo "tmp" >> .gitignore` +4. (Optional) Use the `tree` command to check that your directory structure + looks correct. Running `tree -a` in your `devchallenge` folder should display + a structure like this: + ``` + devchallenge + ├── .gitignore + └── tmp + ├── state.json + └── output.json + ``` + +::: + 1. Create a file called `hello.js` and write the following code. ```js @@ -140,12 +207,30 @@ openfn test --log debug us send messages to the terminal window. -2. Run the job using the CLI +1. Run the job using the CLI ```sh - openfn hello.js + openfn hello.js -o tmp/output.json + ``` + +
+ View expected output + + ``` + [CLI] ⚠ WARNING: No adaptor provided! + [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: + openfn job.js -a common + [CLI] ✔ Compiled from helo.js + [R/T] ♦ Starting job job-1 + [JOB] ℹ Hello World! + [R/T] ✔ Completed job job-1 in 1ms + [CLI] ✔ State written to tmp/output.json + [CLI] ✔ Finished in 17ms ✨ + ``` +
+ Note that our `console.log` statement was printed as `[JOB] Hello world!`. Using the console like this is helpful for debugging and/or understanding what's happening inside our jobs. @@ -153,14 +238,14 @@ happening inside our jobs. #### 🏆 Challenge: Write a job that prints your name 1. Modify `hello.js` to print your name. -2. Re-run the job by running `openfn hello.js -a http`. +2. Re-run the job by running `openfn hello.js -a http -o tmp/output.json`. 3. Validate that you receive the logs below: ``` [CLI] ✔ Compiled job from hello.js [JOB] ℹ My name is { YourName } [R/T] ✔ Operation 1 complete in 0ms - [CLI] ✔ Writing output to ./output.json + [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 366ms! ✨ ``` @@ -200,18 +285,23 @@ Run `openfn help` to see the full list of CLI arguments. 2. Run the job by running ```sh -openfn getPosts.js -i -a http +openfn getPosts.js -i -a http -o tmp/output.json ``` Since it is our first time using the `http` adaptor, we are installing the adaptor using `-i` argument -3. See expected CLI logs +
+ 3. See expected CLI logs - ``` - [CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ - [R/T] ✔ Operation 1 complete in 1.072s - [JOB] ℹ { +``` + [CLI] ✔ Installing packages... + [CLI] ✔ Installed @openfn/language-http@4.2.8 + [CLI] ✔ Installation complete in 14.555s + [CLI] ✔ Compiled from getPosts.js + [R/T] ♦ Starting job job-1 + GET request succeeded with 200 ✓ + [JOB] ℹ { userId: 1, id: 1, title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', @@ -219,11 +309,14 @@ adaptor using `-i` argument 'suscipit recusandae consequuntur expedita et cum\n' + 'reprehenderit molestiae ut ut quas totam\n' + 'nostrum rerum est autem sunt rem eveniet architecto' - } - [R/T] ✔ Operation 2 complete in 0ms - [CLI] ✔ Writing output to ./output.json - [CLI] ✔ Done in 1.42s! ✨ - ``` + } + [R/T] ✔ Completed job job-1 in 872ms + [CLI] ✔ State written to tmp/output.json + [CLI] ✔ Finished in 15.518s ✨ + +``` + +
#### 🏆 Challenge: Get and inspect data via HTTP @@ -232,14 +325,16 @@ Using the API, get a list of users and print the first user object. 1. Create file called `getUsers.js` and write your operation to fetch the user. -2. Run the job using the OpenFn/cli `openfn getUsers.js -a http`. +2. Run the job using the OpenFn/cli + `openfn getUsers.js -a http -o tmp/output.json`. 3. Validate that you receive this expected CLI logs: ```sh -openfn getUsers.js -a http +openfn getUsers.js -a http -o tmp/output.json ``` -3. Validate that you receive this expected CLI logs: +
+See expected CLI logs: ``` [CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ @@ -265,9 +360,11 @@ openfn getUsers.js -a http } } [R/T] ✔ Operation 2 complete in 2ms -[CLI] ✔ Writing output to ./output.json [CLI] ✔ Done in 950ms! ✨ +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ ``` +
+ ### 3. Understanding `state` If a job expression is a set of instructions for a chef (a recipe?) then the @@ -329,7 +426,7 @@ Or you can specify the path to the state file by passing the option -s, Specify a path to your `state.json` file with this command: ```sh -openfn hello.js -a http -s tmp/state.json +openfn hello.js -a http -s tmp/state.json -o tmp/output.json ``` Expected CLI logs @@ -339,7 +436,7 @@ Expected CLI logs GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 876ms [R/T] ✔ Operation 2 complete in 0ms -[CLI] ✔ Writing output to ./output.json +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 1.222s! ✨ ``` @@ -389,7 +486,7 @@ of how to set up `state.configuration` for `language-http`. 3. Now run the job using the following command ```sh - openfn getPosts.js -a http -s tmp/state.json + openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json ``` And validate that you see the expected CLI logs: @@ -408,7 +505,7 @@ of how to set up `state.configuration` for `language-http`. 'nostrum rerum est autem sunt rem eveniet architecto' } [R/T] ✔ Operation 2 complete in 0ms - [CLI] ✔ Writing output to ./output.json + [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 470ms! ✨ ``` @@ -527,7 +624,7 @@ GET request succeeded with 200 ✓ //All of posts for userId 1 ] [R/T] ✔ Operation 3 complete in 12ms -[CLI] ✔ Writing output to ./output.json +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 1.239s! ✨ ``` @@ -612,7 +709,7 @@ GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 722ms [JOB] ℹ [Function (anonymous)] [R/T] ✔ Operation 2 complete in 1ms -[CLI] ✔ Writing output to ./output.json +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 1.102s ✨ ``` @@ -679,7 +776,7 @@ Notice how this code uses the `each` function, a helper function defined in but accessed from this job that is using language-http. Most adaptors import and export many functions from `language-common`. -##### Run **openfn job.js -a http** +##### Run **openfn job.js -a http -o tmp/output.json** > Expected CLI logs @@ -693,7 +790,7 @@ GET request succeeded with 200 ✓ // Posts ] [R/T] ✔ Operation 4 complete in 10ms -[CLI] ✔ Writing output to output.json +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 1.091s! ✨ ``` @@ -792,3 +889,7 @@ openfn hello.js -a http --no-strict-output Learn more about CLI [github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli) + +``` + +``` From e7cf9971a865d236c72c666d4ea15abf5ebd9811 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Wed, 10 May 2023 21:20:00 +0300 Subject: [PATCH 02/17] wip: workflow execution --- docs/cli.md | 72 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 63 insertions(+), 9 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 7407eb2fb2a..4bd52476ec7 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -238,7 +238,7 @@ happening inside our jobs. #### 🏆 Challenge: Write a job that prints your name 1. Modify `hello.js` to print your name. -2. Re-run the job by running `openfn hello.js -a http -o tmp/output.json`. +2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. 3. Validate that you receive the logs below: ``` @@ -492,7 +492,7 @@ of how to set up `state.configuration` for `language-http`. And validate that you see the expected CLI logs: ```sh - [CLI] ✔ Compiled job from job.js + [CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 120ms [JOB] ℹ { @@ -736,7 +736,7 @@ We often have to perform the same operation multiple times for items in an array. Most of the helper functions for data manipulation are inherited from @openfn/language-common and are available in most of the adaptors. -##### Create job.js and add the following codes +##### Modify getPosts.js to group posts by user-ID ```js // Get all posts @@ -776,12 +776,12 @@ Notice how this code uses the `each` function, a helper function defined in but accessed from this job that is using language-http. Most adaptors import and export many functions from `language-common`. -##### Run **openfn job.js -a http -o tmp/output.json** +##### Run **openfn getPosts.js -a http -o tmp/output.json** > Expected CLI logs ```sh -[CLI] ✔ Compiled job from job.js +[CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 730ms [R/T] ✔ Operation 2 complete in 0ms @@ -806,6 +806,64 @@ build function that will get posts by user id. Discuss the results with your administrator. +### 8. Using Execution Plan + +Execution Plan is a powerful feature of `@openfn/cli` that allows you to define +a list of jobs and rules for executing them. You can use an Execution Plan to +orchestrate the flow of data between systems, and to handle errors and retries +in a structured and automated way. + +##### Workflow Plan Structure + +A Workflow Plan is a JSON object that consists of the following properties: + +- `start` (required): The ID of the job that should be executed first. +- `jobs` (required): An array of job objects, each of which represents a + specific task to be executed. + - `id` (required): A unique ID that identifies the job. + - `data` (optional): An object that contains any data that should be passed to + the job. + - `expression` (required): A string that contains a JavaScript function to be + executed as the job. The function should accept a state parameter and return + a new state object. + - `next` (optional): An object that specifies the next job to be executed + based on the output of the current job. The object should have one or more + key-value pairs, where the key is the ID of the next job, and the value is a + boolean expression that determines whether the next job should be executed. + +###### Example of workflow Execution plan + +Here's an example of a simple Workflow Plan that consists of three jobs: + +```json +{ + "start": "start", + "jobs": [ + { + "id": "start", + "data": { + "defaultAnswer": 42 + }, + "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", + "next": { + "calculate": "!state.error" + } + }, + { + "id": "calculate", + "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", + "next": { + "result": true + } + }, + { + "id": "result", + "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" + } + ] +} +``` + ## CLI Usage - Key Commands You’ll learn about these commands in the following challenges, but please refer @@ -889,7 +947,3 @@ openfn hello.js -a http --no-strict-output Learn more about CLI [github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli) - -``` - -``` From e5625fedd0e568f526f22192f6ade89054c410e4 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Thu, 11 May 2023 10:40:46 +0300 Subject: [PATCH 03/17] wip: execution plan --- docs/cli.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 4bd52476ec7..102bcd7e3ed 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -274,7 +274,7 @@ Run `openfn help` to see the full list of CLI arguments. 1. Create a file called `getPosts.js` and write the following code - ```jsx + ```jsx title=getPosts.js get('https://jsonplaceholder.typicode.com/posts'); fn(state => { console.log(state.data[0]); @@ -458,7 +458,7 @@ of how to set up `state.configuration` for `language-http`. 1. Update your `state.json` to look like this: - ```json + ```json title=state.json { "configuration": { "baseUrl": "https://jsonplaceholder.typicode.com" @@ -738,7 +738,7 @@ array. Most of the helper functions for data manipulation are inherited from ##### Modify getPosts.js to group posts by user-ID -```js +```js title="getPosts.js" // Get all posts get('posts'); @@ -808,14 +808,14 @@ Discuss the results with your administrator. ### 8. Using Execution Plan -Execution Plan is a powerful feature of `@openfn/cli` that allows you to define +Execution plan is a powerful feature of `@openfn/cli` that allows you to define a list of jobs and rules for executing them. You can use an Execution Plan to orchestrate the flow of data between systems, and to handle errors and retries in a structured and automated way. ##### Workflow Plan Structure -A Workflow Plan is a JSON object that consists of the following properties: +A workflow plan is a JSON object that consists of the following properties: - `start` (required): The ID of the job that should be executed first. - `jobs` (required): An array of job objects, each of which represents a @@ -833,9 +833,9 @@ A Workflow Plan is a JSON object that consists of the following properties: ###### Example of workflow Execution plan -Here's an example of a simple Workflow Plan that consists of three jobs: +Here's an example of a simple workflow plan that consists of three jobs: -```json +```json title="workflow.json" { "start": "start", "jobs": [ @@ -864,6 +864,12 @@ Here's an example of a simple Workflow Plan that consists of three jobs: } ``` +To execute the workflow execution plan + +``` +openfn workflow.json +``` + ## CLI Usage - Key Commands You’ll learn about these commands in the following challenges, but please refer From eb700bbc3e30a4700d284ce1876fe93a416cb880 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Tue, 16 May 2023 20:19:33 +0300 Subject: [PATCH 04/17] wip: execution plan --- docs/cli.md | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 102bcd7e3ed..a0c1301a79f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -821,11 +821,14 @@ A workflow plan is a JSON object that consists of the following properties: - `jobs` (required): An array of job objects, each of which represents a specific task to be executed. - `id` (required): A unique ID that identifies the job. + - `configuration`: (optional) Specifies the configuration file associated with + the job - `data` (optional): An object that contains any data that should be passed to the job. - - `expression` (required): A string that contains a JavaScript function to be - executed as the job. The function should accept a state parameter and return - a new state object. + - `adaptor` (required): Specifies the adaptor used for the job + - `expression` (required): Specifies the JavaScript file associated with the + job. It can also be a string that contains a JavaScript function to be + executed as the job. - `next` (optional): An object that specifies the next job to be executed based on the output of the current job. The object should have one or more key-value pairs, where the key is the ID of the next job, and the value is a @@ -842,23 +845,28 @@ Here's an example of a simple workflow plan that consists of three jobs: { "id": "start", "data": { - "defaultAnswer": 42 + "name": "Foo Bar" }, - "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", + "adaptor": "common", + "expression": "hello.js", "next": { - "calculate": "!state.error" + "getUsers": "!state.error" } }, { - "id": "calculate", - "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", + "id": "getUsers", + "adaptor": "http", + "expression": "getUsers.js", + "configuration": "http-creds.json", "next": { - "result": true + "getPosts": true } }, { - "id": "result", - "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" + "id": "getPosts", + "adaptor": "http", + "configuration": "http-creds.json", + "expression": "getPosts.js" } ] } From ccb6c23b3153a7351c49340688317f4269a5136b Mon Sep 17 00:00:00 2001 From: mtuchi Date: Wed, 17 May 2023 10:26:00 +0300 Subject: [PATCH 05/17] wip: execution plan with adaptor autoinstall option --- docs/cli.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/cli.md b/docs/cli.md index a0c1301a79f..bb343d39fe7 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -878,6 +878,12 @@ To execute the workflow execution plan openfn workflow.json ``` +To execute the workflow execution plan with adaptor autoinstall option + +``` +openfn workflow.json -i +``` + ## CLI Usage - Key Commands You’ll learn about these commands in the following challenges, but please refer From 9acdb4ec7efbcc73679d75a70fc23347bb72523e Mon Sep 17 00:00:00 2001 From: mtuchi Date: Fri, 19 May 2023 10:26:34 +0300 Subject: [PATCH 06/17] fin: execution plan --- docs/cli.md | 71 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 55 insertions(+), 16 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index bb343d39fe7..d0f5634c6d9 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -836,7 +836,8 @@ A workflow plan is a JSON object that consists of the following properties: ###### Example of workflow Execution plan -Here's an example of a simple workflow plan that consists of three jobs: +
+Here's an example of a simple workflow plan that consists of three jobs: ```json title="workflow.json" { @@ -857,7 +858,7 @@ Here's an example of a simple workflow plan that consists of three jobs: "id": "getUsers", "adaptor": "http", "expression": "getUsers.js", - "configuration": "http-creds.json", + "configuration": "tmp/http-creds.json", "next": { "getPosts": true } @@ -865,14 +866,35 @@ Here's an example of a simple workflow plan that consists of three jobs: { "id": "getPosts", "adaptor": "http", - "configuration": "http-creds.json", + "configuration": "tmp/http-creds.json", "expression": "getPosts.js" } ] } ``` -To execute the workflow execution plan +
+ +To execute the workflow execution plan we run `openfn [path/to/workflow.json]`. + +
+ +For example if you created workflow.json in root of your project directory + + +``` + devchallenge + ├── .gitignore + ├── hello.js + ├── getUsers.js + ├── getPosts.js + ├── workflow.json + └── tmp + ├── http-creds.json + └── output.json +``` + +
``` openfn workflow.json @@ -884,6 +906,35 @@ To execute the workflow execution plan with adaptor autoinstall option openfn workflow.json -i ``` +:::danger Important + +When working with the `workflow.json` file, it is important to handle sensitive +information, such as credentials and initial input data, in a secure manner. To +ensure the protection of your sensitive data, please follow the guidelines +outlined below: + +1. Configuration Key: In the `workflow.json` file, specify a path to a + gitignored configuration file that will be contain necessary credentials that + will be used to access the destination system. For example: + + ``` + { + ... + "configuration": "tmp/openMRS-credentials.json" + }, + ``` + +2. Data Key: Incase you need to path initial data to your job, Specify a path to + a gitignored data file + ``` + { + ... + "data": "tmp/initial-data.json", + } + ``` + +::: + ## CLI Usage - Key Commands You’ll learn about these commands in the following challenges, but please refer @@ -953,17 +1004,5 @@ openfn compile [path] Will compile the openfn job and print or save the resulting js. - - Learn more about CLI [github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli) From 7388b53e5d747c347652a14dc589642e02cbb2c2 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Fri, 19 May 2023 10:43:51 +0300 Subject: [PATCH 07/17] minor improvements --- docs/cli.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index d0f5634c6d9..6c43f9b0c0e 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -241,13 +241,13 @@ happening inside our jobs. 2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. 3. Validate that you receive the logs below: - ``` - [CLI] ✔ Compiled job from hello.js - [JOB] ℹ My name is { YourName } - [R/T] ✔ Operation 1 complete in 0ms - [CLI] ✔ Writing output to tmp/output.json - [CLI] ✔ Done in 366ms! ✨ - ``` +``` +[CLI] ✔ Compiled job from hello.js +[JOB] ℹ My name is { YourName } +[R/T] ✔ Operation 1 complete in 0ms +[CLI] ✔ Writing output to tmp/output.json +[CLI] ✔ Done in 366ms! ✨ +``` ### 2. Using adaptor helper functions From 246495e8e1c4b188b0c500f59f1f987c1aa289c8 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Tue, 23 May 2023 09:39:10 +0300 Subject: [PATCH 08/17] wip fix ci build --- docs/cli.md | 89 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 44 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 6c43f9b0c0e..dc807421773 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -72,7 +72,7 @@ please see the documentation for **[@openfn/core](/documentation/core)** and Let's start by running a simple command with the CLI. Type the following into your terminal: -``` +```bash openfn test ``` @@ -82,7 +82,7 @@ command.
You should see some output like this: -```sh +```bash [CLI] ℹ Versions: ▸ node.js 18.12.1 ▸ cli 0.0.39 @@ -161,7 +161,7 @@ export default [fn()]; You can see this (and a lot more detail) by running the test command with debug-level logging: -``` +```bash openfn test --log debug ``` @@ -185,7 +185,7 @@ openfn test --log debug 4. (Optional) Use the `tree` command to check that your directory structure looks correct. Running `tree -a` in your `devchallenge` folder should display a structure like this: - ``` + ```bash devchallenge ├── .gitignore └── tmp @@ -202,34 +202,35 @@ openfn test --log debug ```
- What is console.log? - console.log is a core JavaScript language function which lets - us send messages to the terminal window. + What is console.log? + console.log is a core JavaScript language function which lets + us send messages to the terminal window.
1. Run the job using the CLI - ```sh - openfn hello.js -o tmp/output.json - ``` + ```bash + openfn hello.js -o tmp/output.json + ``` + +
-
View expected output - ``` - [CLI] ⚠ WARNING: No adaptor provided! - [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: - openfn job.js -a common - [CLI] ✔ Compiled from helo.js - [R/T] ♦ Starting job job-1 - [JOB] ℹ Hello World! - [R/T] ✔ Completed job job-1 in 1ms - [CLI] ✔ State written to tmp/output.json - [CLI] ✔ Finished in 17ms ✨ + ```bash + [CLI] ⚠ WARNING: No adaptor provided! + [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: + openfn job.js -a common + [CLI] ✔ Compiled from helo.js + [R/T] ♦ Starting job job-1 + [JOB] ℹ Hello World! + [R/T] ✔ Completed job job-1 in 1ms + [CLI] ✔ State written to tmp/output.json + [CLI] ✔ Finished in 17ms ✨ - ``` + ``` -
+
Note that our `console.log` statement was printed as `[JOB] Hello world!`. Using the console like this is helpful for debugging and/or understanding what's @@ -241,7 +242,7 @@ happening inside our jobs. 2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. 3. Validate that you receive the logs below: -``` +```bash [CLI] ✔ Compiled job from hello.js [JOB] ℹ My name is { YourName } [R/T] ✔ Operation 1 complete in 0ms @@ -284,7 +285,7 @@ Run `openfn help` to see the full list of CLI arguments. 2. Run the job by running -```sh +```bash openfn getPosts.js -i -a http -o tmp/output.json ``` @@ -294,7 +295,7 @@ adaptor using `-i` argument
3. See expected CLI logs -``` +```bash [CLI] ✔ Installing packages... [CLI] ✔ Installed @openfn/language-http@4.2.8 [CLI] ✔ Installation complete in 14.555s @@ -329,7 +330,7 @@ API, get a list of users and print the first user object. `openfn getUsers.js -a http -o tmp/output.json`. 3. Validate that you receive this expected CLI logs: -```sh +```bash openfn getUsers.js -a http -o tmp/output.json ``` @@ -425,7 +426,7 @@ Or you can specify the path to the state file by passing the option -s, Specify a path to your `state.json` file with this command: -```sh +```bash openfn hello.js -a http -s tmp/state.json -o tmp/output.json ``` @@ -485,13 +486,13 @@ of how to set up `state.configuration` for `language-http`. 3. Now run the job using the following command - ```sh + ```bash openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json ``` And validate that you see the expected CLI logs: - ```sh + ```bash [CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 120ms @@ -679,7 +680,7 @@ fn(state => { > Expected CLI logs -``` +```bash [CLI] ✘ TypeError: path.match is not a function at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26) at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22) @@ -703,7 +704,7 @@ fix the error by passing a string in dataValue i.e `console.log(dataValue(“1 > Expected CLI logs -``` +```bash [CLI] ✔ Compiled job from debug.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 722ms @@ -780,7 +781,7 @@ export many functions from `language-common`. > Expected CLI logs -```sh +```bash [CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 730ms @@ -882,7 +883,7 @@ To execute the workflow execution plan we run `openfn [path/to/workflow.json]`. For example if you created workflow.json in root of your project directory -``` +```bash devchallenge ├── .gitignore ├── hello.js @@ -896,13 +897,13 @@ For example if you created workflow.json in root of your project di
-``` +```bash openfn workflow.json ``` To execute the workflow execution plan with adaptor autoinstall option -``` +```bash openfn workflow.json -i ``` @@ -917,7 +918,7 @@ outlined below: gitignored configuration file that will be contain necessary credentials that will be used to access the destination system. For example: - ``` + ```json { ... "configuration": "tmp/openMRS-credentials.json" @@ -926,7 +927,7 @@ outlined below: 2. Data Key: Incase you need to path initial data to your job, Specify a path to a gitignored data file - ``` + ```json { ... "data": "tmp/initial-data.json", @@ -942,19 +943,19 @@ to this section for the key commands used in working with the CLI. ### Check the version -``` +```bash openfn version ``` ### Get help -``` +```bash openfn help ``` ### Run a job -``` +```bash openfn path/to/job.js -ia {adaptor-name} ``` @@ -967,7 +968,7 @@ You can find the list of publicly available adaptors [here](/adaptors). > file) For example `openfn execute hello.js ` Reads hello.js, looks for state > and output in foo -``` +```bash -i, --autoinstall Auto-install the language adaptor -a, --adaptors, --adaptor A language adaptor to use for the job ``` @@ -981,7 +982,7 @@ You can pass `-l info` or `--log info` to get more feedback about what's happening, or `--log debug` for more details than you could ever use. Below is the list of different log levels -``` +```bash openfn hello.js -a http -l none ``` @@ -998,7 +999,7 @@ The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust, portable, and easier to debug from a pure JS perspective. -``` +```bash openfn compile [path] ``` From 12e03c75b8a66d140868c67e5340c17df6740a23 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Wed, 14 Jun 2023 16:14:00 +0300 Subject: [PATCH 09/17] wip: update workflow execution plan --- docs/cli.md | 110 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 90 insertions(+), 20 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index dc807421773..03c536350cb 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -814,26 +814,37 @@ a list of jobs and rules for executing them. You can use an Execution Plan to orchestrate the flow of data between systems, and to handle errors and retries in a structured and automated way. +_For example, if you have two jobs in your workflow (GET users from system A & +POST users to system B), you can use your execution plan to automatically run +both jobs from start to finish. This imitates the +[flow and fail trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) +on the OpenFn platform where a second job should run after the first one +succeeds or fails, respectively, using the data returned from the first job. “_ + ##### Workflow Plan Structure A workflow plan is a JSON object that consists of the following properties: -- `start` (required): The ID of the job that should be executed first. +- `start` (optional): The ID of the job that should be executed first (defaults + to jobs[0]). - `jobs` (required): An array of job objects, each of which represents a specific task to be executed. - - `id` (required): A unique ID that identifies the job. + - `id` (required): A job name that is unique to the workflow and helps you ID + your job. - `configuration`: (optional) Specifies the configuration file associated with the job - - `data` (optional): An object that contains any data that should be passed to - the job. - - `adaptor` (required): Specifies the adaptor used for the job + - `data` (optional): An object that contains any pre-populate data that should + be passed to the job (this will be overriden by keys in previous state). + - `adaptor` (required): Specifies the adaptor used for the job (version + optional) - `expression` (required): Specifies the JavaScript file associated with the job. It can also be a string that contains a JavaScript function to be executed as the job. - - `next` (optional): An object that specifies the next job to be executed - based on the output of the current job. The object should have one or more - key-value pairs, where the key is the ID of the next job, and the value is a - boolean expression that determines whether the next job should be executed. + - `next` (optional): An object that specifies which jobs to call next.All + edges returning true will run. The object should have one or more key-value + pairs, where the key is the ID of the next job, and the value is a boolean + expression that determines whether the next job should be executed.If there + are no next edges, the workflow will end ###### Example of workflow Execution plan @@ -842,18 +853,13 @@ A workflow plan is a JSON object that consists of the following properties: ```json title="workflow.json" { - "start": "start", + "start": "getUsers", "jobs": [ { - "id": "start", - "data": { - "name": "Foo Bar" - }, - "adaptor": "common", - "expression": "hello.js", - "next": { - "getUsers": "!state.error" - } + "id": "createUsers", + "adaptor": "http", + "expression": "createUsers.js", + "configuration": "tmp/http-creds.json" }, { "id": "getUsers", @@ -861,7 +867,8 @@ A workflow plan is a JSON object that consists of the following properties: "expression": "getUsers.js", "configuration": "tmp/http-creds.json", "next": { - "getPosts": true + "createUsers": true, + "getPosts": false } }, { @@ -876,6 +883,64 @@ A workflow plan is a JSON object that consists of the following properties:
+
+ tmp/http-creds.json + +```json title="http-creds.json" +{ + "baseUrl": "https://jsonplaceholder.typicode.com/" +} +``` + +
+ +
+ getUsers.js + +```json title="getUsers.js" +get('users', {}, state => { + function changeEmailDomain(email) { + // Split the email into username and domain parts + const [username, domain] = email.split('@'); + return `${username}@openf.demo`; + } + + const newUsers = state.data.map(user => { + return { + name: `${user.name}-OpenFn`, + username: `${user.username}-openfn`, + email: changeEmailDomain(user.email), + website: 'openfn.org', + company: { + name: 'OpenFn', + catchPhrase: 'Integration and automation made easy', + }, + }; + }); + return { ...state, data: {}, response: {}, newUsers }; +}); +``` + +
+ +
+ createUsers.js + +```json title="createUsers.js" +post('users', state => state.newUsers); +``` + +
+ +
+ getPosts.js + +```json title="getPosts.js" +get('posts'); +``` + +
+ To execute the workflow execution plan we run `openfn [path/to/workflow.json]`.
@@ -888,6 +953,7 @@ For example if you created workflow.json in root of your project di ├── .gitignore ├── hello.js ├── getUsers.js + ├── createUsers.js ├── getPosts.js ├── workflow.json └── tmp @@ -907,6 +973,10 @@ To execute the workflow execution plan with adaptor autoinstall option openfn workflow.json -i ``` +_On execution, this workflow plan will first auto-install the adaptors then run +*getUsers* job successed then *createUsers* will run using the final state of +*getUsers*. getPosts will not run because it was specified as false_ + :::danger Important When working with the `workflow.json` file, it is important to handle sensitive From 24c37a36471f460a30f939f438bcacbd51cf36f0 Mon Sep 17 00:00:00 2001 From: mtuchi Date: Wed, 14 Jun 2023 18:09:15 +0300 Subject: [PATCH 10/17] improve running workflow --- docs/cli.md | 56 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 38 insertions(+), 18 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 03c536350cb..cab73d23f4c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -807,23 +807,31 @@ build function that will get posts by user id. Discuss the results with your administrator. -### 8. Using Execution Plan +### 8. Running Workflows -Execution plan is a powerful feature of `@openfn/cli` that allows you to define -a list of jobs and rules for executing them. You can use an Execution Plan to -orchestrate the flow of data between systems, and to handle errors and retries -in a structured and automated way. +As of `v0.0.35` the `@openfn/cli` supports running workflows as well as jobs. +that allows you to define a list of jobs and rules for executing them. You can +use a workflow to orchestrate the flow of data between systems in a structured +and automated way. _For example, if you have two jobs in your workflow (GET users from system A & -POST users to system B), you can use your execution plan to automatically run -both jobs from start to finish. This imitates the +POST users to system B), you can setup your workflow to run all jobs in sequence +from start to finish. This imitates the [flow and fail trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) on the OpenFn platform where a second job should run after the first one succeeds or fails, respectively, using the data returned from the first job. “_ -##### Workflow Plan Structure +:::info TLRD -A workflow plan is a JSON object that consists of the following properties: +You won't have to assemble the initial state of the next job, the final state of +the upstream job will be passed down to the down stream job as initial state + +::: + +##### Workflow + +A workflow is in execution plan for running several jobs in a sequence. It is +defined as a JSON object that consists of the following properties: - `start` (optional): The ID of the job that should be executed first (defaults to jobs[0]). @@ -846,10 +854,10 @@ A workflow plan is a JSON object that consists of the following properties: expression that determines whether the next job should be executed.If there are no next edges, the workflow will end -###### Example of workflow Execution plan +###### Example of a workflow
-Here's an example of a simple workflow plan that consists of three jobs: +Here's an example of a simple workflow that consists of three jobs: ```json title="workflow.json" { @@ -941,7 +949,7 @@ get('posts');
-To execute the workflow execution plan we run `openfn [path/to/workflow.json]`. +To execute the workflow we run `openfn [path/to/workflow.json]`.
@@ -964,18 +972,30 @@ For example if you created workflow.json in root of your project di
```bash -openfn workflow.json +openfn workflow.json -o tmp/output.json ``` -To execute the workflow execution plan with adaptor autoinstall option +:::info This will work only if adaptors are installed + +On execution, this workflow will first run _getUsers_ job,If successed then +_createUsers_ will run using the final state of _getUsers_. _getPosts_ will not +run + +::: + +To execute the workflow with adaptor autoinstall option ```bash -openfn workflow.json -i +openfn workflow.json -i -o tmp/output.json ``` -_On execution, this workflow plan will first auto-install the adaptors then run -*getUsers* job successed then *createUsers* will run using the final state of -*getUsers*. getPosts will not run because it was specified as false_ +:::info + +On execution, this workflow will first auto-install the adaptors then run +_getUsers_ job,If successed then _createUsers_ will run using the final state of +_getUsers_. _getPosts_ will not run + +::: :::danger Important From f0e50321d7be196cb16c358a7382e58b3a0eaccd Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 26 Jun 2023 15:06:49 +0300 Subject: [PATCH 11/17] improve running workflow --- docs/cli.md | 197 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 143 insertions(+), 54 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index cab73d23f4c..d30c6bc4879 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -817,9 +817,9 @@ and automated way. _For example, if you have two jobs in your workflow (GET users from system A & POST users to system B), you can setup your workflow to run all jobs in sequence from start to finish. This imitates the -[flow and fail trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) +[flow trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) on the OpenFn platform where a second job should run after the first one -succeeds or fails, respectively, using the data returned from the first job. “_ +succeeds, respectively, using the data returned from the first job. “_ :::info TLRD @@ -861,29 +861,30 @@ defined as a JSON object that consists of the following properties: ```json title="workflow.json" { - "start": "getUsers", + "start": "getPatients", "jobs": [ { - "id": "createUsers", + "id": "getPatients", "adaptor": "http", - "expression": "createUsers.js", - "configuration": "tmp/http-creds.json" + "expression": "getPatients.js", + "configuration": "tmp/http-creds.json", + "next": { + "getGlobalOrgsUnit": true + } }, { - "id": "getUsers", - "adaptor": "http", - "expression": "getUsers.js", - "configuration": "tmp/http-creds.json", + "id": "getGlobalOrgsUnit", + "adaptor": "common", + "expression": "getGlobalOrgsUnit.js", "next": { - "createUsers": true, - "getPosts": false + "createTEIs": true } }, { - "id": "getPosts", - "adaptor": "http", - "configuration": "tmp/http-creds.json", - "expression": "getPosts.js" + "id": "createTEIs", + "adaptor": "dhis2", + "expression": "createTEIs.js", + "configuration": "tmp/dhis2-creds.json" } ] } @@ -894,7 +895,7 @@ defined as a JSON object that consists of the following properties:
tmp/http-creds.json -```json title="http-creds.json" +```json title="tmp/http-creds.json" { "baseUrl": "https://jsonplaceholder.typicode.com/" } @@ -903,48 +904,136 @@ defined as a JSON object that consists of the following properties:
- getUsers.js - -```json title="getUsers.js" -get('users', {}, state => { - function changeEmailDomain(email) { - // Split the email into username and domain parts - const [username, domain] = email.split('@'); - return `${username}@openf.demo`; - } + tmp/dhis2-creds.json - const newUsers = state.data.map(user => { - return { - name: `${user.name}-OpenFn`, - username: `${user.username}-openfn`, - email: changeEmailDomain(user.email), - website: 'openfn.org', - company: { - name: 'OpenFn', - catchPhrase: 'Integration and automation made easy', - }, - }; - }); - return { ...state, data: {}, response: {}, newUsers }; +```json title="tmp/dhis2-creds.json" +{ + "hostUrl": "https://play.dhis2.org/2.39.1.2", + "password": "district", + "username": "admin" +} +``` + +
+ +
+ getPatients.js + +```js title="getPatients.js" +// Get users from jsonplaceholder +get('users'); + +// Prepare new users as new patients +fn(state => { + const newPatients = state.data; + return { ...state, newPatients }; }); ```
- createUsers.js + getGlobalOrgsUnit.js -```json title="createUsers.js" -post('users', state => state.newUsers); +```js title="getGlobalOrgsUnit.js" +// Globals: orgUnits +fn(state => { + const globalOrgUnits = [ + { + label: 'Njandama MCHP', + id: 'g8upMTyEZGZ', + source: 'Gwenborough', + }, + { + label: 'Njandama MCHP', + id: 'g8upMTyEZGZ', + source: 'Wisokyburgh', + }, + { + label: 'Njandama MCHP', + id: 'g8upMTyEZGZ', + source: 'McKenziehaven', + }, + { + label: 'Njandama MCHP', + id: 'g8upMTyEZGZ', + source: 'South Elvis', + }, + { + label: 'Ngelehun CHC', + id: 'IpHINAT79UW', + source: 'Roscoeview', + }, + { + label: 'Ngelehun CHC', + id: 'IpHINAT79UW', + source: 'South Christy', + }, + { + label: 'Ngelehun CHC', + id: 'IpHINAT79UW', + source: 'Howemouth', + }, + { + label: 'Ngelehun CHC', + id: 'IpHINAT79UW', + source: 'Aliyaview', + }, + { + label: 'Baoma Station CHP', + id: 'jNb63DIHuwU', + source: 'Bartholomebury', + }, + { + label: 'Baoma Station CHP', + id: 'jNb63DIHuwU', + source: 'Lebsackbury', + }, + ]; + + return { ...state, globalOrgUnits }; +}); ```
- getPosts.js + createTEIs.js -```json title="getPosts.js" -get('posts'); +```js title="createTEIs.js" +fn(state => { + const { newPatients, globalOrgUnits } = state; + + const getOrgUnit = city => + globalOrgUnits.find(orgUnit => orgUnit.source === city).id; + + const mappedEntities = newPatients.map(patient => { + const [firstName = 'Patient', lastName = 'Test'] = ( + patient.name || '' + ).split(' '); + + const orgUnit = getOrgUnit(patient.address.city); + + const attributes = [ + { attribute: 'w75KJ2mc4zz', value: firstName }, + { attribute: 'zDhUuAYrxNC', value: lastName }, + { attribute: 'cejWyOfXge6', value: 'Male' }, + ]; + + return { ...patient, attributes: attributes, orgUnit: orgUnit }; + }); + + return { ...state, mappedEntities }; +}); + +each( + 'mappedEntities[*]', + create('trackedEntityInstances', { + orgUnit: dataValue('orgUnit'), + trackedEntityType: 'nEenWmSyUEp', + attributes: dataValue('attributes'), + }) +); ```
@@ -959,13 +1048,13 @@ For example if you created workflow.json in root of your project di ```bash devchallenge ├── .gitignore - ├── hello.js - ├── getUsers.js - ├── createUsers.js - ├── getPosts.js + ├── getPatients.js + ├── createTEIs.js + ├── getGlobalOrgsUnit.js ├── workflow.json └── tmp ├── http-creds.json + ├── dhis2-creds.json └── output.json ``` @@ -977,9 +1066,9 @@ openfn workflow.json -o tmp/output.json :::info This will work only if adaptors are installed -On execution, this workflow will first run _getUsers_ job,If successed then -_createUsers_ will run using the final state of _getUsers_. _getPosts_ will not -run +On execution, this workflow will first run _getPatients_ job,If successed then +_createTEIs_ will run using the final state of _getPatients_. +_getGlobalOrgsUnit_ will not run ::: @@ -992,8 +1081,8 @@ openfn workflow.json -i -o tmp/output.json :::info On execution, this workflow will first auto-install the adaptors then run -_getUsers_ job,If successed then _createUsers_ will run using the final state of -_getUsers_. _getPosts_ will not run +_getPatients_ job,If successed then _createTEIs_ will run using the final state +of _getPatients_. _getGlobalOrgsUnit_ will not run ::: From 4119b051fdc9d2545c8afa2003c93234599154f4 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 26 Jun 2023 15:19:21 +0300 Subject: [PATCH 12/17] grammar fixes --- docs/cli.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index d30c6bc4879..c0aec313066 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -810,12 +810,12 @@ Discuss the results with your administrator. ### 8. Running Workflows As of `v0.0.35` the `@openfn/cli` supports running workflows as well as jobs. -that allows you to define a list of jobs and rules for executing them. You can +that allow you to define a list of jobs and rules for executing them. You can use a workflow to orchestrate the flow of data between systems in a structured and automated way. _For example, if you have two jobs in your workflow (GET users from system A & -POST users to system B), you can setup your workflow to run all jobs in sequence +POST users to system B), you can set up your workflow to run all jobs in sequence from start to finish. This imitates the [flow trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) on the OpenFn platform where a second job should run after the first one @@ -824,13 +824,13 @@ succeeds, respectively, using the data returned from the first job. “_ :::info TLRD You won't have to assemble the initial state of the next job, the final state of -the upstream job will be passed down to the down stream job as initial state +the upstream job will be passed down to the downstream job as initial state ::: ##### Workflow -A workflow is in execution plan for running several jobs in a sequence. It is +A workflow is in the execution plan for running several jobs in a sequence. It is defined as a JSON object that consists of the following properties: - `start` (optional): The ID of the job that should be executed first (defaults @@ -842,7 +842,7 @@ defined as a JSON object that consists of the following properties: - `configuration`: (optional) Specifies the configuration file associated with the job - `data` (optional): An object that contains any pre-populate data that should - be passed to the job (this will be overriden by keys in previous state). + be passed to the job (this will be overridden by keys in the previous state). - `adaptor` (required): Specifies the adaptor used for the job (version optional) - `expression` (required): Specifies the JavaScript file associated with the @@ -1066,7 +1066,7 @@ openfn workflow.json -o tmp/output.json :::info This will work only if adaptors are installed -On execution, this workflow will first run _getPatients_ job,If successed then +On execution, this workflow will first run _getPatients_ job,If succeed then _createTEIs_ will run using the final state of _getPatients_. _getGlobalOrgsUnit_ will not run @@ -1081,7 +1081,7 @@ openfn workflow.json -i -o tmp/output.json :::info On execution, this workflow will first auto-install the adaptors then run -_getPatients_ job,If successed then _createTEIs_ will run using the final state +_getPatients_ job,If succeed then _createTEIs_ will run using the final state of _getPatients_. _getGlobalOrgsUnit_ will not run ::: @@ -1094,7 +1094,7 @@ ensure the protection of your sensitive data, please follow the guidelines outlined below: 1. Configuration Key: In the `workflow.json` file, specify a path to a - gitignored configuration file that will be contain necessary credentials that + git ignored configuration file that will contain necessary credentials that will be used to access the destination system. For example: ```json From ac1a139fd3b98766a4bc596be6990a1ecaa3a10f Mon Sep 17 00:00:00 2001 From: Taylor Downs Date: Tue, 27 Jun 2023 08:56:46 +0100 Subject: [PATCH 13/17] cleanup --- docs/cli.md | 52 +++++++++++++++++++++++----------------------------- 1 file changed, 23 insertions(+), 29 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index c0aec313066..ebac0680e04 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -815,13 +815,13 @@ use a workflow to orchestrate the flow of data between systems in a structured and automated way. _For example, if you have two jobs in your workflow (GET users from system A & -POST users to system B), you can set up your workflow to run all jobs in sequence -from start to finish. This imitates the +POST users to system B), you can set up your workflow to run all jobs in +sequence from start to finish. This imitates the [flow trigger patterns](https://docs.openfn.org/documentation/build/triggers#flow-triggers) on the OpenFn platform where a second job should run after the first one succeeds, respectively, using the data returned from the first job. “_ -:::info TLRD +:::info tl;dr You won't have to assemble the initial state of the next job, the final state of the upstream job will be passed down to the downstream job as initial state @@ -830,8 +830,8 @@ the upstream job will be passed down to the downstream job as initial state ##### Workflow -A workflow is in the execution plan for running several jobs in a sequence. It is -defined as a JSON object that consists of the following properties: +A workflow is in the execution plan for running several jobs in a sequence. It +is defined as a JSON object that consists of the following properties: - `start` (optional): The ID of the job that should be executed first (defaults to jobs[0]). @@ -842,7 +842,8 @@ defined as a JSON object that consists of the following properties: - `configuration`: (optional) Specifies the configuration file associated with the job - `data` (optional): An object that contains any pre-populate data that should - be passed to the job (this will be overridden by keys in the previous state). + be passed to the job (this will be overridden by keys in the previous + state). - `adaptor` (required): Specifies the adaptor used for the job (version optional) - `expression` (required): Specifies the JavaScript file associated with the @@ -869,13 +870,13 @@ defined as a JSON object that consists of the following properties: "expression": "getPatients.js", "configuration": "tmp/http-creds.json", "next": { - "getGlobalOrgsUnit": true + "getGlobalOrgUnits": true } }, { - "id": "getGlobalOrgsUnit", + "id": "getGlobalOrgUnits", "adaptor": "common", - "expression": "getGlobalOrgsUnit.js", + "expression": "getGlobalOrgUnits.js", "next": { "createTEIs": true } @@ -933,9 +934,9 @@ fn(state => {
- getGlobalOrgsUnit.js + getGlobalOrgUnits.js -```js title="getGlobalOrgsUnit.js" +```js title="getGlobalOrgUnits.js" // Globals: orgUnits fn(state => { const globalOrgUnits = [ @@ -1050,7 +1051,7 @@ For example if you created workflow.json in root of your project di ├── .gitignore ├── getPatients.js ├── createTEIs.js - ├── getGlobalOrgsUnit.js + ├── getGlobalOrgUnits.js ├── workflow.json └── tmp ├── http-creds.json @@ -1064,27 +1065,20 @@ For example if you created workflow.json in root of your project di openfn workflow.json -o tmp/output.json ``` -:::info This will work only if adaptors are installed +On execution, this workflow will first run the `getPatients` job,If succeed then +`createTEIs` will run using the final state of `getPatients`. +`getGlobalOrgUnits` will not run. -On execution, this workflow will first run _getPatients_ job,If succeed then -_createTEIs_ will run using the final state of _getPatients_. -_getGlobalOrgsUnit_ will not run - -::: - -To execute the workflow with adaptor autoinstall option +Note that without the `-i` flag, you'll need to already have your adaptor +installed. To execute the workflow with adaptor autoinstall option: ```bash openfn workflow.json -i -o tmp/output.json ``` -:::info - On execution, this workflow will first auto-install the adaptors then run -_getPatients_ job,If succeed then _createTEIs_ will run using the final state -of _getPatients_. _getGlobalOrgsUnit_ will not run - -::: +`getPatients` job. If that first job succeeds then `createTEIs` will run using +the final state of `getPatients`. The `getGlobalOrgUnits` job will not run. :::danger Important @@ -1093,9 +1087,9 @@ information, such as credentials and initial input data, in a secure manner. To ensure the protection of your sensitive data, please follow the guidelines outlined below: -1. Configuration Key: In the `workflow.json` file, specify a path to a - git ignored configuration file that will contain necessary credentials that - will be used to access the destination system. For example: +1. Configuration Key: In the `workflow.json` file, specify a path to a git + ignored configuration file that will contain necessary credentials that will + be used to access the destination system. For example: ```json { From 92e38fd1bd092c0994e816c8bc85aefe43bfc90c Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 27 Jun 2023 12:48:24 +0300 Subject: [PATCH 14/17] update on execution description --- docs/cli.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index ebac0680e04..433c3dd2cd8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1066,8 +1066,9 @@ openfn workflow.json -o tmp/output.json ``` On execution, this workflow will first run the `getPatients` job,If succeed then -`createTEIs` will run using the final state of `getPatients`. -`getGlobalOrgUnits` will not run. +`getGlobalOrgUnits` will run using the final state of `getPatients`. If +`getGlobalOrgUnits` succeed then `createTEIs` will run using the final state of +`getGlobalOrgUnits`. Note that without the `-i` flag, you'll need to already have your adaptor installed. To execute the workflow with adaptor autoinstall option: @@ -1077,8 +1078,9 @@ openfn workflow.json -i -o tmp/output.json ``` On execution, this workflow will first auto-install the adaptors then run -`getPatients` job. If that first job succeeds then `createTEIs` will run using -the final state of `getPatients`. The `getGlobalOrgUnits` job will not run. +`getPatients` job,If succeed then `getGlobalOrgUnits` will run using the final +state of `getPatients`. If `getGlobalOrgUnits` succeed then `createTEIs` will +run using the final state of `getGlobalOrgUnits`. :::danger Important From 869d950c62f5b95c48e78ea45cfbd402eee5498a Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 27 Jun 2023 15:51:11 +0300 Subject: [PATCH 15/17] grammar fixes --- docs/cli.md | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 433c3dd2cd8..5ca009f0218 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -810,7 +810,7 @@ Discuss the results with your administrator. ### 8. Running Workflows As of `v0.0.35` the `@openfn/cli` supports running workflows as well as jobs. -that allow you to define a list of jobs and rules for executing them. You can +This allow you to define a list of jobs and rules for executing them. You can use a workflow to orchestrate the flow of data between systems in a structured and automated way. @@ -824,14 +824,15 @@ succeeds, respectively, using the data returned from the first job. “_ :::info tl;dr You won't have to assemble the initial state of the next job, the final state of -the upstream job will be passed down to the downstream job as initial state +the upstream job will automatically be passed down to the downstream job as the +initial state ::: ##### Workflow -A workflow is in the execution plan for running several jobs in a sequence. It -is defined as a JSON object that consists of the following properties: +A workflow is the execution plan for running several jobs in a sequence. It is +defined as a JSON object that consists of the following properties: - `start` (optional): The ID of the job that should be executed first (defaults to jobs[0]). @@ -840,20 +841,18 @@ is defined as a JSON object that consists of the following properties: - `id` (required): A job name that is unique to the workflow and helps you ID your job. - `configuration`: (optional) Specifies the configuration file associated with - the job - - `data` (optional): An object that contains any pre-populate data that should - be passed to the job (this will be overridden by keys in the previous - state). + the job. + - `data` (optional): An object that contains any pre-populated data. - `adaptor` (required): Specifies the adaptor used for the job (version - optional) + optional). - `expression` (required): Specifies the JavaScript file associated with the job. It can also be a string that contains a JavaScript function to be executed as the job. - - `next` (optional): An object that specifies which jobs to call next.All + - `next` (optional): An object that specifies which jobs to call next. All edges returning true will run. The object should have one or more key-value pairs, where the key is the ID of the next job, and the value is a boolean expression that determines whether the next job should be executed.If there - are no next edges, the workflow will end + are no next edges, the workflow will end. ###### Example of a workflow @@ -1039,11 +1038,11 @@ each(
-To execute the workflow we run `openfn [path/to/workflow.json]`. +Run `openfn [path/to/workflow.json]` to execute the workflow.
-For example if you created workflow.json in root of your project directory +For example if you created workflow.json in the root of your project directory, This is how your project will look like ```bash @@ -1065,22 +1064,21 @@ For example if you created workflow.json in root of your project di openfn workflow.json -o tmp/output.json ``` -On execution, this workflow will first run the `getPatients` job,If succeed then -`getGlobalOrgUnits` will run using the final state of `getPatients`. If -`getGlobalOrgUnits` succeed then `createTEIs` will run using the final state of -`getGlobalOrgUnits`. +On execution, this workflow will first run the `getPatients.js` job. If is +successful then `getGlobalOrgUnits.js` will run using the final state of +`getPatients.js`. If `getGlobalOrgUnits.js` is successful then `createTEIs.js` +will run using the final state of `getGlobalOrgUnits.js`. Note that without the `-i` flag, you'll need to already have your adaptor -installed. To execute the workflow with adaptor autoinstall option: +installed. To execute the workflow with the adaptor autoinstall option run this +command: ```bash openfn workflow.json -i -o tmp/output.json ``` -On execution, this workflow will first auto-install the adaptors then run -`getPatients` job,If succeed then `getGlobalOrgUnits` will run using the final -state of `getPatients`. If `getGlobalOrgUnits` succeed then `createTEIs` will -run using the final state of `getGlobalOrgUnits`. +On execution, this workflow will first auto-install the adaptors then run the +workflow :::danger Important @@ -1100,7 +1098,7 @@ outlined below: }, ``` -2. Data Key: Incase you need to path initial data to your job, Specify a path to +2. Data Key: Incase you need to pass initial data to your job, specify a path to a gitignored data file ```json { From 231e4f26d1305557d71b133fac353481fcaa8cd8 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 27 Jun 2023 16:04:45 +0300 Subject: [PATCH 16/17] improvements --- docs/cli.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 5ca009f0218..62156ff1e15 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -809,10 +809,10 @@ Discuss the results with your administrator. ### 8. Running Workflows -As of `v0.0.35` the `@openfn/cli` supports running workflows as well as jobs. -This allow you to define a list of jobs and rules for executing them. You can -use a workflow to orchestrate the flow of data between systems in a structured -and automated way. +As of `v0.0.35` the `@openfn/cli` supports running not only jobs, but also +_workflows_. Running a workflow allows you to define a list of jobs and rules +for executing them. You can use a workflow to orchestrate the flow of data +between systems in a structured and automated way. _For example, if you have two jobs in your workflow (GET users from system A & POST users to system B), you can set up your workflow to run all jobs in @@ -825,7 +825,7 @@ succeeds, respectively, using the data returned from the first job. “_ You won't have to assemble the initial state of the next job, the final state of the upstream job will automatically be passed down to the downstream job as the -initial state +initial state. ::: @@ -842,7 +842,7 @@ defined as a JSON object that consists of the following properties: your job. - `configuration`: (optional) Specifies the configuration file associated with the job. - - `data` (optional): An object that contains any pre-populated data. + - `data` (optional): A JSON object that contains the pre-populated data. - `adaptor` (required): Specifies the adaptor used for the job (version optional). - `expression` (required): Specifies the JavaScript file associated with the @@ -1065,9 +1065,9 @@ openfn workflow.json -o tmp/output.json ``` On execution, this workflow will first run the `getPatients.js` job. If is -successful then `getGlobalOrgUnits.js` will run using the final state of -`getPatients.js`. If `getGlobalOrgUnits.js` is successful then `createTEIs.js` -will run using the final state of `getGlobalOrgUnits.js`. +successful, `getGlobalOrgUnits.js` will run using the final state of +`getPatients.js`. If `getGlobalOrgUnits.js` is successful, `createTEIs.js` will +run using the final state of `getGlobalOrgUnits.js`. Note that without the `-i` flag, you'll need to already have your adaptor installed. To execute the workflow with the adaptor autoinstall option run this From ebb557237f5f08554b82f33b0f5acaaec6f6716b Mon Sep 17 00:00:00 2001 From: Aicha Diallo Date: Sun, 2 Jul 2023 22:33:05 +0000 Subject: [PATCH 17/17] FCA considerations --- adaptors/salesforce.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/adaptors/salesforce.md b/adaptors/salesforce.md index f6735dc1b6a..d4bddc16570 100644 --- a/adaptors/salesforce.md +++ b/adaptors/salesforce.md @@ -146,6 +146,12 @@ values (e.g., `M: male, F: female`). semicolon-separated strings (e.g., `Services__c: 'Food;Counselling;Medical_Aid`). +#### Timeout considerations + +The Salesforce adaptor currently waits for the batch to finish syncing to Salesforce to ensure that the batch results (i.e., whether the batch import was successful, how many records succeeded, how many failures) can be relayed back to the OpenFn user in Activity History. This can potentially cause timeout errors in OpenFn if the batch sync triggers Salesforce automation which takes long to complete. + +In future adaptor updates, OpenFn will consider adding a new option to the adaptor to not wait for the batch results, and to have the batch run asynchronously. The tradeoff is that the OpenFn run would always succeed–and the administrator would have to log into Salesforce and monitor the batch results via the Salesforce Setup “Bulk Data Load Jobs” page. With this approach, OpenFn wouldn’t set limits on the batch processing time avoiding timeout errors, but it does move the batch monitoring from OpenFn to Salesforce. + ### Salesforce Credentials Salesforce requires a username, password, login URL, and security token to @@ -207,7 +213,8 @@ Please save this `security token` in your OpenFn `Credential`. 8. `INVALID_FIELD_FOR_INSERT_UPDATE: Unable to create/update fields`: This might occur when trying to update a Relationship field, for example a Person related to a Person's Visit. The field setting `Allow reparenting` on the - Master-Detail relationship field may need to be turned on . + Master-Detail relationship field may need to be turned on. +9. `UNABLE_TO_LOCK_ROW: unable to obtain exclusive access to this record`: This error occurs when either 1) the OpenFn job tries to update the same record more than once at the same time or 2) the OpenFn job tries to updates a Salesforce record at the same time as someone else in the Salesforce system (this includes any automation that may be running in parallel to the OpenFn jobs). ## OpenFn Adaptors