From a803a96df693e0f663faf5d76542eb967afdeb6b Mon Sep 17 00:00:00 2001 From: Sergey Motornyuk Date: Wed, 13 Mar 2024 01:22:42 +0200 Subject: [PATCH] chore: update docs --- README.md | 348 +++++++++++++++++- ckanext/files/assets/scripts/files--shared.js | 33 +- ckanext/files/assets/ts/files--shared.ts | 40 +- screenshots/1.png | Bin 0 -> 18827 bytes screenshots/2.png | Bin 0 -> 24040 bytes screenshots/3.png | Bin 0 -> 20105 bytes screenshots/4.png | Bin 0 -> 26853 bytes setup.cfg | 4 +- 8 files changed, 391 insertions(+), 34 deletions(-) create mode 100644 screenshots/1.png create mode 100644 screenshots/2.png create mode 100644 screenshots/3.png create mode 100644 screenshots/4.png diff --git a/README.md b/README.md index ce3e628..9bdc782 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ To install ckanext-files: pip install ckanext-files # Google Cloud Storage support - pip install 'ckanext-files[gcstorage]' + pip install 'ckanext-files[gcs]' ``` 1. Add `files` to the `ckan.plugins` setting in your CKAN @@ -43,16 +43,284 @@ To install ckanext-files: 1. Run DB migrations ```sh - # CKAN >= v2.9 with alembic + # CKAN >= v2.9 ckan db upgrade -p files # CKAN == v2.8 - paster --plugin=ckanext-files files -c ckan.ini initdb + paster --plugin=ckanext-files files -c ckan.ini initdb ``` ## Usage +### Configure the storage +Before uploading any file, you have to configure a named **storage**. Tell +extension which driver to use(i.e, where and how data will be stored), add few +storage specific settings and you are ready to go. Let's start from the Redis +driver, because it has minimal requirements in terms of configuration. + +Add the following line to the CKAN config file: + +```ini +ckanext.files.storage.my_storage.type = files:redis +``` + +Look at this option. The prefix `ckanext.files.storage.` will be the same for +every configuration option related to storage. Then comes `my_storage`. It's a +name of your storage. It doesn't change the behavior of the storage, but +determines how options are grouped together and helps you to use multiple +storages simultaneously. For example the following configuration adds two +different storages:`first_storage` with `prefix` set to `first:` and +`second_storage` with prefix set to `second:`: + +```ini +ckanext.files.storage.first_storage.type = files:redis +ckanext.files.storage.second_storage.type = files:redis + +ckanext.files.storage.first_storage.prefix = first: +ckanext.files.storage.second_storage.prefix = second: +``` + +Our storage is called `my_storage` and after the storage's name comes option's +name `type`. And it's value `files:redis`. `type` option tells us which driver +we are going to use for the storage. `files:redis` is the name of the driver. + +The recommended format of the driver name is +`:`. But this format is not enforced, so you can +see any value there. If you accidentally make a typo in the driver's name, +starting from v2.10 CKAN will show you an error message on startup with the +list of available drivers: + +```sh +Invalid configuration values provided: +ckanext.files.storage.my_storage.type: Value must be one of ['files:bq_google_cloud_storage', 'files:fs', 'files:public_fs', 'files:redis', 'files:google_cloud_storage'] +Aborted! +``` + +Storage is configured, so we can actually upload the file. Let's use +[ckanapi](https://github.com/ckan/ckanapi) for this task. Files are created via +`files_file_create` API action and this time we have to pass 3 parameters into it: + +* `name`: the name of uploaded file +* `upload`: content of the file +* `storage`: name of the storage that stores the file(we just configured `my_storage`) + +The final command is here: + +```sh +ckanapi action files_file_create \ + name=hello.txt \ + upload='hello world' \ + storage=my_storage +``` + +And that's what you see as result: + +```json +{ + "atime": null, + "completed": true, + "ctime": "2024-03-12T22:08:05.185914", + "id": "7f9a7676-5177-40f0-b610-0b47918fdccc", + "mtime": null, + "name": "hello.txt", + "storage": "my_storage", + "storage_data": { + "content_type": "text/plain", + "filename": "2f51aeff-96a5-4d79-8973-7867527d7f2e", + "hash": "5eb63bbbe01eeed093cb22bb8f5acdc3", + "size": 11 + } +} +``` + +Now go to Redis CLI and check the content of the file. Note, you cannot get the +content via CKAN API, because it's JSON-based and downloading files doesn't +suit its principles. + +By default, Redis driver puts the content under the key +``. Pay attention to `FILENAME`. It's the value available as +`storage_data.filename` in the API response(i.e, +`2f51aeff-96a5-4d79-8973-7867527d7f2e` in our case), not the name of the real +file you just uploaded. + +`PREFIX` can be configured, but we skipped this step and got the default value: `ckanext:files:default:file_content:`. So the final Redis key of our file is `ckanext:files:default:file_content:2f51aeff-96a5-4d79-8973-7867527d7f2e` + +```sh +redis-cli + +127.0.0.1:6379> GET ckanext:files:default:file_content:2f51aeff-96a5-4d79-8973-7867527d7f2e +"hello world" + +``` + +And before we moved further, let's remove the file, using ID we got when it was created: + +```sh +ckanapi action files_file_delete id=7f9a7676-5177-40f0-b610-0b47918fdccc + +# ... response from api + +redis-cli + +127.0.0.1:6379> GET ckanext:files:default:file_content:2f51aeff-96a5-4d79-8973-7867527d7f2e +(nil) + +``` + +### Usage in code + +If you are writing the code and you want to interact with the storage directly, +without the API layer, you can do it via a number of public functions of the +extension. + +Let's configure filesystem storage first. This time we are going to use the +storage name `default`. This is "default" storage name - if you haven't +specified explicitely the name of the storage when working with it(calling API +actions, for example), `default` storage is used. So it's better to create this +storage instead of making fancy names. + +Filesystem driver has a mandatory option `path` that controls path, where files +are stored. If path does not exist, storage will raise an exception by +default. But it can also create missing path if you enable `create_path` +option. Here's our final version of settings: + +```ini +ckanext.files.storage.default.type = files:fs +ckanext.files.storage.default.path = /tmp/example +ckanext.files.storage.default.create_path = true +``` + +Now we are going to connect to CKAN shell via `ckan shell` CLI command and +create an instance of the storage: + +```python +from ckanext.files.shared import get_storage +storage = get_storage('default') +``` + +Because you have all configuration in place, the rest is fairly +straightforward. We will upload the file, read it's content and remove it from +the CKAN shell. + +Uploading is the most challenging step. As CKAN is based on Flask, we have to +upload files as Flask does. For it we are going to create an instance of `werkzeug.datastructures.FileStorage` and pass bytes stream to its constructor. After that, we can pass the object into storage's `upload` method and specify the name of the upload. + +```python +from werkzeug.datastructures import FileStorage +from io import BytesIO + +upload = FileStorage(BytesIO(b'hello world')) +result = storage.upload('file.txt', upload, {}) + +print(result) + +... {'filename': 'd65da0fd-299a-433c-850e-086fdc5ebb7e', +... 'content_type': None, +... 'size': 11, +... 'hash': '5eb63bbbe01eeed093cb22bb8f5acdc3'} + +``` + +`result` contains minimal amount of information that is required by storage to +manage the file. In case of FS storage, it includes `filename`. If you visit +`/tmp/example` directory, that we specified as a `path` for our storage, you'll +see there a file with the name matching `filename` from result. And its content +matches the content of our upload, which is quite an expected outcome. + +But let's go back to the shell and try reading file from the python's +code. We'll pass `result` to the storage's `stream` method, which produces a +readable buffer based on our result. This buffer has `read` method so we can +work with the buffer just as we usually do with IO streams: + +```python +buffer = storage.stream(result) +content = buffer.read() +print(content) + +... b'hello world' +``` + +And finally we need to remove the file + +```python +storage.remove(result) +``` + +### Usage in browser + +You can upload files using JavaScript CKAN modules. The extension extends +CKAN's Sandbox object(available as `this.sandbox` inside the JS CKAN module), +so we can use shortcut and upload file directly from the DevTools. Open any +CKAN page, switch to JS console and create the sandbox instance. Inside it we +have `files` object, which in turn contains `upload` method. This method +accepts `File` object for upload(the same object you can get from the +`input[type=file]`). + +```js +sandbox = ckan.sandbox() +await sandbox.files.upload( + new File(["content"], "file.txt") +) + +... { +... "id": "d6fa3096-613a-4b16-a0eb-b7a6ec91d690", +... "name": "file.txt", +... "storage": "default", +... "ctime": "2024-03-12T23:08:11.291971", +... "mtime": null, +... "atime": null, +... "storage_data": { +... "filename": "f484d116-cf6f-4238-80e5-905849873be0", +... "content_type": "application/octet-stream", +... "size": 7, +... "hash": "9a0364b9e99bb480dd25e1f0284c8555" +... }, +... "completed": true +... } +``` + +If you are still using FS storage configured in previous section, switch to +`/tmp/example` folder and check it's content: + +```sh +ls /tmp/example +... f484d116-cf6f-4238-80e5-905849873be0 + +cat f484d116-cf6f-4238-80e5-905849873be0 +... content +``` + +And, as usually, let's remove file using the ID from the `upload` promise: + +```js +sandbox.client.call('POST', 'files_file_delete', { + id: 'd6fa3096-613a-4b16-a0eb-b7a6ec91d690' +}) +``` + +### UI + +This extension provides a basic UI as an example and it's recommended to build +your own UI if you need it. But we can still use default functionality for +testing. Open the `/user//files` page of CKAN application in the +browser. + +![Empty files page](screenshots/1.png) + +Choose the file + +![Selected file](screenshots/2.png) + +Click on play button(the one with triangle icon) and wait a bit. If nothing +changes, check JS console, it can show some errors when storage is not properly +configured. + +![Uploaded file](screenshots/3.png) + +When upload finished, reload the page to see uploaded file. + +![Reloaded page with a single upload](screenshots/4.png) ## Configuration @@ -87,5 +355,79 @@ ckanext.files.default_storage = default # (optional, default: ) ckanext.files.storage..