Skip to content

Commit

Permalink
Merge pull request #2 from DataShades/storage
Browse files Browse the repository at this point in the history
Storage
  • Loading branch information
smotornyuk authored Jun 12, 2024
2 parents eb5a8ba + 83bb811 commit ce04756
Show file tree
Hide file tree
Showing 95 changed files with 9,661 additions and 484 deletions.
13 changes: 13 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# EditorConfig http://EditorConfig.org

# Project Root
root = true

# Default Code Style
[*]
charset = utf-8
end_of_line = lf
indent_size = 4
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true
32 changes: 21 additions & 11 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,18 +1,30 @@
name: Tests
on: [push, pull_request]
on:
push:
branches:
- master
pull_request:
branches:
- master

jobs:
test:
strategy:
matrix:
ckan-version: ["2.10"]
fail-fast: false

runs-on: ubuntu-latest
container:
# The CKAN version tag of the Solr and Postgres containers should match
# the one of the container the tests run on.
# You can switch this base image with a custom image tailored to your project
image: openknowledge/ckan-dev:2.9
image: openknowledge/ckan-dev:${{ matrix.ckan-version }}
services:
solr:
image: ckan/ckan-solr-dev:2.9
image: ckan/ckan-solr:${{ matrix.ckan-version }}
postgres:
image: ckan/ckan-postgres-dev:2.9
image: ckan/ckan-postgres-dev:${{ matrix.ckan-version }}
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
Expand All @@ -29,20 +41,18 @@ jobs:
CKAN_REDIS_URL: redis://redis:6379/1

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: Install requirements
# Install any extra requirements your extension has here (dev requirements, other extensions etc)
run: |
pip install -r requirements.txt
pip install -r dev-requirements.txt
pip install -e .
pip install -e ".[dev,gcs]"
- name: Setup extension
# Extra initialization steps
run: |
# Replace default path to CKAN core config file with the one on the container
sed -i -e 's/use = config:.*/use = config:\/srv\/app\/src\/ckan\/test-core.ini/' test.ini
- name: Setup extension
run: |
ckan -c test.ini db init
- name: Run tests
run: pytest --ckan-ini=test.ini --cov=ckanext.files --disable-warnings ckanext/files

run: pytest --ckan-ini=test.ini --cov=ckanext.files --disable-warnings ckanext -p no:rerunfailures
38 changes: 38 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
default_install_hook_types:
- pre-commit
- pre-push
- commit-msg

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
# - id: check-yaml
- id: end-of-file-fixer
stages: [pre-commit]
- id: trailing-whitespace
stages: [pre-commit]
- id: debug-statements
stages: [pre-push]

## Isort
- repo: https://github.com/pycqa/isort
rev: 5.13.2
hooks:
- id: isort
name: isort
stages: [pre-commit]

## Black
- repo: https://github.com/psf/black
rev: 24.2.0
hooks:
- id: black
stages: [pre-commit]

## Ruff
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: v0.3.0
hooks:
- id: ruff
stages: [pre-commit]
139 changes: 137 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,149 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
<!-- insertion marker -->
## Unreleased

<small>[Compare with latest](https://github.com/DataShades/ckanext-files/compare/v0.0.2...HEAD)</small>
<small>[Compare with latest](https://github.com/DataShades/ckanext-files/compare/v0.3.1...HEAD)</small>

### Features

- optional hash verification for multipart upload ([28e5f69](https://github.com/DataShades/ckanext-files/commit/28e5f6952295afae401faa18fde9fdbb509211e8) by Sergey Motornyuk).
- add supported_types option for storages to restict upload types ([c5b43ac](https://github.com/DataShades/ckanext-files/commit/c5b43acabe11e2d5c94f5137df5948b0b29ea00c) by Sergey Motornyuk).
- add files_file_search action ([b8e8b4c](https://github.com/DataShades/ckanext-files/commit/b8e8b4c638ce4ae1159f47e7ccd62f021550b1e5) by Sergey Motornyuk).
- File.get method ([591ec48](https://github.com/DataShades/ckanext-files/commit/591ec48d4043caf486570faa4af586b11c31f6e9) by Sergey Motornyuk).
- get_storage without arguments returns default storage ([571e021](https://github.com/DataShades/ckanext-files/commit/571e021c44c4d719431c05a837bd6336b1896249) by Sergey Motornyuk).
- use timezone-aware date columns in model ([ae91cc7](https://github.com/DataShades/ckanext-files/commit/ae91cc79ae3b5e52098232fa9dc294c72942ea0d) by Sergey Motornyuk).

### Code Refactoring

- add completed flag for rename, show and delete actions for simultaneous file and multipart support. ([10fb202](https://github.com/DataShades/ckanext-files/commit/10fb202a6e8c95dceee9a878a869812a84353219) by Sergey Motornyuk).
- rename files_upload_show to files_multipart_refresh ([ee2a4df](https://github.com/DataShades/ckanext-files/commit/ee2a4dfdddad4009636590a28b1d664dc321e42e) by Sergey Motornyuk).
- add kwargs to all Storage methods and extras to all service methods ([1526df1](https://github.com/DataShades/ckanext-files/commit/1526df10f566f02f70d1f0b88369fa5b69a3a815) by Sergey Motornyuk).
- rename Storage and Uploader *_multipart_upload into multipart_* for consistency with actions ([ddfd111](https://github.com/DataShades/ckanext-files/commit/ddfd11170286f463fe5115103b7357db106a6802) by Sergey Motornyuk).
- rename files_upload_* actions to files_multipart_*(initialize changed to start) ([6493a1d](https://github.com/DataShades/ckanext-files/commit/6493a1d5cb8225346d5414914d73a8ca3b9276b7) by Sergey Motornyuk).
- rename MULTIPART_UPLOAD capability to MULTIPART ([20d01bf](https://github.com/DataShades/ckanext-files/commit/20d01bf32eb52a91f734627664a2ab265238dcdb) by Sergey Motornyuk).
- use custom dataclass for Upload instead of werkzeug.datastructures.FileStorage ([78ae63b](https://github.com/DataShades/ckanext-files/commit/78ae63b79d63c095313af6e90a5c583f6d0678d6) by Sergey Motornyuk).
- move hash, size, location(former filename) and content_type to the top level of file entity ([45a2679](https://github.com/DataShades/ckanext-files/commit/45a2679498a874ca6c2a00d154dd73a0bc394b29) by Sergey Motornyuk).
- extract File.completed==False into Multipart model ([d90d786](https://github.com/DataShades/ckanext-files/commit/d90d78684829291976a71384a49b40e66234386b) by Sergey Motornyuk).
- use dataclasses instead of dict in storage ([4965568](https://github.com/DataShades/ckanext-files/commit/4965568e22e7a36672c96a26e88e73c21f056730) by Sergey Motornyuk).
- storage_from_settings renamed to make_storage ([08fd767](https://github.com/DataShades/ckanext-files/commit/08fd76751ba1cb778a4c819613388be25bc099c6) by Sergey Motornyuk).
- transform combine_capabilities and exclude_capabilities into Capability methods ([73d32d4](https://github.com/DataShades/ckanext-files/commit/73d32d4d9f18b7e2b75fbf985eb29e73eed4183b) by Sergey Motornyuk).
- replace CapabilityCluster and CapabilityUnit with Capability ([16d3b7e](https://github.com/DataShades/ckanext-files/commit/16d3b7e39cee430e4ae2d2d48256a70827b4c26d) by Sergey Motornyuk).
- remove re-imported types from ckanext.files.types ([4b9e870](https://github.com/DataShades/ckanext-files/commit/4b9e870dcced1d3bffc39719dec8afcada96a913) by Sergey Motornyuk).
- remove support of CKAN pre v2.10 ([3e70bc2](https://github.com/DataShades/ckanext-files/commit/3e70bc27440a9c21560c50e3c744b247f2087e90) by Sergey Motornyuk).
- UnsupportedOperationError constructed with adapter type instead of name ([55d038d](https://github.com/DataShades/ckanext-files/commit/55d038d52e3dcfc09fb259122456079a197f4be8) by Sergey Motornyuk).

<!-- insertion marker -->
## [v0.3.1](https://github.com/DataShades/ckanext-files/releases/tag/v0.3.1) - 2024-05-22

<small>[Compare with v0.3.0](https://github.com/DataShades/ckanext-files/compare/v0.3.0...v0.3.1)</small>

### Features

- generic_download view ([d000446](https://github.com/DataShades/ckanext-files/commit/d0004464f12ba76aac2531f33dad72247b1a62ca) by Sergey Motornyuk).

## [v0.3.0](https://github.com/DataShades/ckanext-files/releases/tag/v0.3.0) - 2024-05-16

<small>[Compare with v0.0.6](https://github.com/DataShades/ckanext-files/compare/v0.0.6...v0.3.0)</small>

### Features

- files_uploader plugin compatible with native uploader interface ([31aaaa6](https://github.com/DataShades/ckanext-files/commit/31aaaa676c3f1a0aba2bb3a706f85deb066895fa) by Sergey Motornyuk).

### Bug Fixes

- upload errors rendered outside of view box ([48005ed](https://github.com/DataShades/ckanext-files/commit/48005ed4229110dfca43fb219ba2bff4b8c9f5ba) by Sergey Motornyuk).
- upload errors in actions not tracked ([530c6d9](https://github.com/DataShades/ckanext-files/commit/530c6d98dcdb3e923c8eb2639cfef36e1b5e6d42) by Sergey Motornyuk).

### Code Refactoring

- disallow file creation via auth function ([0db289b](https://github.com/DataShades/ckanext-files/commit/0db289bfbbc3de99c3b49fbc671009db4406ccff) by Sergey Motornyuk).

## [v0.0.6](https://github.com/DataShades/ckanext-files/releases/tag/v0.0.6) - 2024-04-24

<small>[Compare with v0.2.6](https://github.com/DataShades/ckanext-files/compare/v0.2.6...v0.0.6)</small>

### Bug Fixes

- declarations are missing from the package ([15fa97b](https://github.com/DataShades/ckanext-files/commit/15fa97b4c9fdaf6211f3e74e9cbf71eb19166a6b) by Sergey Motornyuk).

## [v0.2.6](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.6) - 2024-04-24

<small>[Compare with v0.2.4](https://github.com/DataShades/ckanext-files/compare/v0.2.4...v0.2.6)</small>

### Bug Fixes

- declarations are missing from the package ([16c4999](https://github.com/DataShades/ckanext-files/commit/16c499945740facc9f8f1c301fc088bbc78a81ab) by Sergey Motornyuk).
- catch permission error on delete ([9e6d799](https://github.com/DataShades/ckanext-files/commit/9e6d799417ec842cfea5b671446a91657e5fd6c9) by Sergey Motornyuk).

## [v0.2.4](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.4) - 2024-04-15

<small>[Compare with v0.2.3](https://github.com/DataShades/ckanext-files/compare/v0.2.3...v0.2.4)</small>

### Features

- add dropzone and immediate upload ([0486a00](https://github.com/DataShades/ckanext-files/commit/0486a007a3eb1178cb8e838160ac84579024fa68) by Sergey Motornyuk).

## [v0.2.3](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.3) - 2024-04-07

<small>[Compare with v0.2.2](https://github.com/DataShades/ckanext-files/compare/v0.2.2...v0.2.3)</small>

### Features

- file search by plugin data ([9dc51bd](https://github.com/DataShades/ckanext-files/commit/9dc51bd9f67f58d3f77aeff0247e9eb224ea0a38) by Sergey Motornyuk).
- multipart uploaders accept initialize/complete payloads in JS ([97d9933](https://github.com/DataShades/ckanext-files/commit/97d9933f69dd4fe4053912c75ba3db41e44c34e2) by Sergey Motornyuk).

### Bug Fixes

- python2 fails when content-length accessed ([6e99315](https://github.com/DataShades/ckanext-files/commit/6e993154d6988d3d144dad0790e9860daa0ab2b6) by Sergey Motornyuk).

## [v0.2.2](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.2) - 2024-03-18

<small>[Compare with v0.2.1](https://github.com/DataShades/ckanext-files/compare/v0.2.1...v0.2.2)</small>

## [v0.2.1](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.1) - 2024-03-18

<small>[Compare with v0.2.0](https://github.com/DataShades/ckanext-files/compare/v0.2.0...v0.2.1)</small>

### Features

- add move and copy operations ([577b537](https://github.com/DataShades/ckanext-files/commit/577b5377474afbdc9293655127dacdd4bc325b5b) by Sergey Motornyuk).

## [v0.2.0](https://github.com/DataShades/ckanext-files/releases/tag/v0.2.0) - 2024-03-12

<small>[Compare with v0.0.5](https://github.com/DataShades/ckanext-files/compare/v0.0.5...v0.2.0)</small>

### Features

- UI for file uploads ([4121e6f](https://github.com/DataShades/ckanext-files/commit/4121e6f530bfe7cf8bd77759a15e9e859886aa7c) by Sergey Motornyuk).
- redis storage ([cced1e8](https://github.com/DataShades/ckanext-files/commit/cced1e898666b14fd4f536405d42800cefa28640) by Sergey Motornyuk).
- multipart upload api ([96051d4](https://github.com/DataShades/ckanext-files/commit/96051d4b8449e6b7f96bf8e05a3860d972624326) by Sergey Motornyuk).
- GCS storage ([cc5ee76](https://github.com/DataShades/ckanext-files/commit/cc5ee76825748ea4988603c409a558d68bcb7434) by Sergey Motornyuk).
- split files into Storage and File ([71a7765](https://github.com/DataShades/ckanext-files/commit/71a7765304486f637a1d34b71c500a1bc8aaae04) by Sergey Motornyuk).

### Code Refactoring

- switch to typescript ([dfb5060](https://github.com/DataShades/ckanext-files/commit/dfb5060e1308c7de9600b0ac4f31aa7ee1bdbc10) by Sergey Motornyuk).
- full type coverage ([f399582](https://github.com/DataShades/ckanext-files/commit/f399582e2e9a6b6d612823b29709f52aaf45e887) by Sergey Motornyuk).
- get rid of blankets ([dfe901c](https://github.com/DataShades/ckanext-files/commit/dfe901c52b07111799793a7af7c37d0f9a024364) by Sergey Motornyuk).
- make types py2 compatible ([c099dfc](https://github.com/DataShades/ckanext-files/commit/c099dfc00534f11ab927406749e6503b509469a7) by Sergey Motornyuk).
- remove ckanext-toolbelt dependency ([dc885c7](https://github.com/DataShades/ckanext-files/commit/dc885c7a36e5f1d2103a58c7a4e5882e40bc7e77) by Sergey Motornyuk).

## [v0.0.5](https://github.com/DataShades/ckanext-files/releases/tag/v0.0.5) - 2024-02-26

<small>[Compare with v0.0.4](https://github.com/DataShades/ckanext-files/compare/v0.0.4...v0.0.5)</small>

### Bug Fixes

- fix auth functions ([c76d5ff](https://github.com/DataShades/ckanext-files/commit/c76d5ffc0bde5731eb820bf9f4fb262965be5120) by mutantsan).
- collection type without generic ([0647355](https://github.com/DataShades/ckanext-files/commit/064735521fc9d704676e40f6199cbc6d1eefd208) by Sergey Motornyuk).

## [v0.0.4](https://github.com/DataShades/ckanext-files/releases/tag/v0.0.4) - 2023-10-25

<small>[Compare with v0.0.2](https://github.com/DataShades/ckanext-files/compare/v0.0.2...v0.0.4)</small>

## [v0.0.2](https://github.com/DataShades/ckanext-files/releases/tag/v0.0.2) - 2022-02-09

<small>[Compare with v0.0.1](https://github.com/DataShades/ckanext-files/compare/v0.0.1...v0.0.2)</small>

## [v0.0.1](https://github.com/DataShades/ckanext-files/releases/tag/v0.0.1) - 2021-09-21

<small>[Compare with first commit](https://github.com/DataShades/ckanext-files/compare/d57d17e412821d56a9f5262636be89311e8050fc...v0.0.1)</small>

3 changes: 3 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
License
+++++++

GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007

Expand Down
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
include README.rst
include LICENSE
include requirements.txt
recursive-include ckanext *.html *.json *.js *.less *.css *.mo *.yml *.yaml
recursive-include ckanext/files *.html *.json *.js *.less *.css *.mo *.yml *.config *.yaml
recursive-include ckanext/file_manager *.html *.json *.js *.less *.css *.mo *.yml *.config *.yaml
recursive-include ckanext/files/migration *.ini *.py *.mako
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ help:


changelog: ## compile changelog
git changelog -c conventional -o CHANGELOG.md
git changelog -c conventional -o CHANGELOG.md $(if $(bump),-B $(bump))
Loading

0 comments on commit ce04756

Please sign in to comment.