Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data and history retention edits #575

Merged
merged 7 commits into from
Oct 18, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 16 additions & 23 deletions docs/manage-projects/io-data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,22 @@ In the "Data Storage" section of your Project `Settings`, you can configure what
you'd like OpenFn to do with the actual _data_ (`Inputs` and `Outputs`) that are
processed (or produced) by your workflow runs.

Users may configure OpenFn to temporarily store data (e.g., forms fetched from
CommCare mobile app) so that transactions can be easily troubleshot and
rectified in case of errors (e.g., destination DHIS2 system is down, or a
database constraint/validation blocks a data import). Data retention periods
have a default time period set by the OpenFn super admin, but can be modified
according to a specific project's requirements.

### Why would I store input/output data along with run logs?

Users may configure OpenFn to temporarily store data (e.g., forms fetched from
CommCare mobile app) so that transactions can be easily troubleshooted and
rectified in case of errors (e.g., destination DHIS2 system is down, or a
database constraint/validation blocks a data import). Data retention periods
have a default time period set by the OpenFn super admin, but can be modified
according to a specific project's requirements.
have a default time period set by the OpenFn instance administrator, but can be
modified according to a specific project's requirements.

One of the most powerful features of the platform is the ability to "replay"
Work Orders. In the scenario where you have a multi-step workflow (e.g., get
work orders. In the scenario where you have a multi-step workflow (e.g., get
data from Database, transform & map the data, and import to your Health
Information System), this temporary data storage allows OpenFn administrators to
quickly troubleshoot failed Work Orders and "re-try" the workflow from the
quickly troubleshoot failed work orders and "re-try" the workflow from the
failed step, rather than re-running the workflow from the very beginning. This
allows administrators to reprocess the failed Work Orders without having to
allows administrators to reprocess the failed work orders without having to
fetch (or re-send) the data (Inputs) from a source system.

### Why would I chose to _NOT_ store input/output data?
Expand All @@ -53,13 +46,13 @@ pipelines.
### Export History

Users can also export all work orders and their associated artefacts (runs,
steps, runsteps, input and output data clips) in a project. Work order
history export is managed at a project level and available to all collaborators (viewer,
steps, runsteps, input and output data clips) in a project. Work Order history
export is managed at a project level and available to all collaborators (viewer,
editor, admin, owner) in a project.

#### How to export work order history

To export Work Order history for your project, open the project and click on
To export work order history for your project, open the project and click on
`History` on the side menu. On the History page, scroll to the bottom of the
work order history table and click on the cloud icon (see image below).

Expand All @@ -70,22 +63,22 @@ If confirmed, a background process will be initiated for the export.

![Confirm export](/img/confirm_export.png)

When the export is complete, an email will be sent to your OpenFn user's associated
email address.
When the export is complete, an email will be sent to your OpenFn user's
associated email address.

:::info FOR LOCAL DEPLOYMENTS

For local deployments, OpenFn uses Swoosh as a mailbox service for development
purposes and can access the mailbox via http://localhost:4000/dev/mailbox. You
can substitute localhost:4000 to match the port hsoting your OpenFn instance.
can substitute localhost:4000 to match the port hosting your OpenFn instance.

:::

#### Managing Exports

Users can view all history exports via the `History Exports`
page in the project settings. Click on `Settings` on the side menu and click on
`History Exports` to see the list of work order exports in your project.
Users can view all history exports via the `History Exports` page in the project
settings. Click on `Settings` on the side menu and click on `History Exports` to
see the list of work order exports in your project.

On the `History Export`s page, you will see the list of exports showing your
latest request and previous requests with the other information such as
Expand All @@ -95,8 +88,8 @@ filename, date of export, user who requested the export and the status.

:::caution Configuring storage for exports

OpenFn instance administrators can configure the storage for work order exports for
local deployments. OpenFn currently supports local storage and Google Cloud
OpenFn instance administrators can configure the storage for work order exports
for local deployments. OpenFn currently supports local storage and Google Cloud
Storage as destinations for exporting work orders.

:::
8 changes: 5 additions & 3 deletions docs/manage-projects/retention-periods.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ month or 90-days) to reduce data storage costs or to limit their data footprint.

### What happens when old work orders, runs, logs, or dataclips get removed?

If you've configured any automatic data exports or archiving, you can still
access your history in those CSVs, but all history will be removed from the
OpenFn platform.
All history is removed from OpenFn and cannot be accessible on the platform. If
you have previously requested
[work order history exports,](https://docs.openfn.org/documentation/manage-projects/io-data-storage?_gl=1*1x5ctqh*_gcl_au*MTE3Njc2MDkyMC4xNzI0MjQzMTAwLjY0NTI0ODAyMC4xNzI3MTY5OTM5LjE3MjcxNjk5NDA.#export-history)
you'll still be able to access exported CSVs via the history export page in
project settings.