From 1b153b0c1ca7a5a22f7215d1379114ae1ef8d4ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Demenech?= Date: Mon, 26 Feb 2024 19:13:23 -0300 Subject: [PATCH] Sync/2024 02 26 (#310) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix dataset collaborator and issue delete authorization * fix(map view): prevent client error when legend config is empty * Feature/pending datasets (#198) * Add pending_datasets table init and CRUD * Add approval_status and draft to metadata * Fix draft permissions * [main.yml] Init. pendingdb tables * Run WRI unit tests first * Fix paths in unit tests script * Run WRI unit tests first in src * Fix OR in unit tests script --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> * Rm test.yml (#202) * [xl]: approval ui implementation * add approval permission * Normalize filenames instead of using a uuid (#205) * approval workflow implementation * add test * fix test * update permission * add issue tab indexing * update approval workflow * update documentation * display filled metadata fields * Prefect datapusher (#208) * Prefect datapusher * Progress * Datapusher * Use default id * Progress * Datapusher ci/cd (#207) * [main.yml] Add CI/CD for datapusher * [values.yaml.dev.template] Add github sha for datapusher * Progress * Progress * Progress * Progress * Fix build * FInisihed * Remove rm command * Fix tests * Fix tests * Wait on prefect * Fix typo * Trigger CI * Change viewport * Update tests * Update github action * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Check logs * Fix tests * Fix tests * Datapusher token * Fix tests * Fix tests * Fix tests * Init exteions * Fix tests * Fix tests * Change permissions of file * Fix tests * Fix test * Trigger CI * Remove api_token * Fix tests * Fix build * Update datapusher * Fix tests * Fix unit tests * Fix test.ini --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> * Return RW Error message (#210) * Return RW Error message * Fix build * Trigger CI * Fix file upload * Fix datapusher and resource upload (#211) * Improvements table view (#212) * Improvements table view * Datepicker on tabular view * Default values (#213) * Default values * Fix pagination * Enable other data formats (#215) * Fix datafiles editing (#216) * Feat/chart view connector url (#219) * feat: basic chart creation functionallity * feat: chart view creation/visualization * feat(chart view): fix self is not defined error, imrpove formats select, responsiveness fixes * fix(chart view): dybamic import chart * fix(chart view): dybamic import chart * feat(chat view): public view options * feat(chat view): implement pie charts * feat(chat view): implement embedding * fix(chart view): minor fixes to the editor * fix(chart view): show all format options * feat(chart view): refactor components structure * feat(chart view): makes it possible to create chart views based on connector url * fix(chart view): linting errors * fix(chart view): chart creation button (#222) * QA Fixes (#223) * Show datapusher status (#225) * Show datapusher status * Status on tab for datapusher * Add spinner to scheduled * Fix/chart view ii (#226) * feat(chart view): allow users to see dataset charts * feat(chart view): dataset views are now embedable * feat(chart view): update resource and view update button label * feat(chart view): show a tooltip with a message when chart view cannot be saved due to pending preview update * fix(chart view): enable tooltips by default * feat(chart view): start implementing chart view icons * fix(chart view): fix page braking when maps are present * fix build * trigger build * temporarily remove test * Fix xlsx datapusher (#227) * Fix datastore query (#228) * Fix/chart view ii (#229) * feat(chart view): allow users to see dataset charts * feat(chart view): dataset views are now embedable * feat(chart view): update resource and view update button label * feat(chart view): show a tooltip with a message when chart view cannot be saved due to pending preview update * fix(chart view): enable tooltips by default * feat(chart view): start implementing chart view icons * fix(chart view): fix page braking when maps are present * fix build * trigger build * temporarily remove test * build: add map test again * build: remove temporary test * feat(chart view): maintain flag on dataset that indicates whether it has chart views or not * fix(chart view): do not allow category, measure and dimension to be the same * fix(map view): fix tests * build: add map test again * fix(map view): fix tests * fix(tests): remove unnecessary file * fix build * feat(chart view): add tests * feat(chart view): add docs * fix: remove unnecessary header from rw api * fix(chart view): proxy requests to rw api (#230) * Tabular preview fixes (#231) * Tabular preview fixes * Add back () * Trigger CI * Fix/chart view iii (#234) * fix(chart view): add loading indicator for view create and update + fix success notification theme * fix(chart view): fix plot height issue when plot is refreshed * fix(chart view): tweak sorting behavior * Docs datapusher tabular preview (#232) * Boot up staging env (#21) * Add staging files * Enable s3filestore and disable init cont. * Initial frontend (#23) --------- Co-authored-by: Luccas Mateus * [hotfix] Fix cert manager name for ingress * [main.yml] Fix frontend build secrets * [cost-splitting] Change the links according to the branch * [cost-splitting] Update branch name * Avoid overwriting cost splitting files of staging * Fix url for cypress tests * Sync dev to staging and enable frontend deployment (#47) * Updating the storage report * Cost splitting will now run on different buckets (#24) Depending on branch * Updating the storage report * Switch to cost splitting branch for s3filestore * Updating the storage report * Updating the storage report * [Dockerfile] Enable s3filestore * [values.yaml.dev.template] Enable s3filestore * Normalize date string * Updating the storage report * Add all files * Updating the storage report * Fix typo * Updating the storage report * Get org title * Updating the storage report * Fix typo * Updating the storage report * Debug * Updating the storage report * Clear stuff * Updating the storage report * Updating the storage report * Initial frontend (#25) Co-authored-by: Luccas Mateus * Rename the base directory as it includes frontend too now * Use datapusher from the ecr * Enable frontend and use images from ECR * Enable frontend build and push * [values.yaml.dev.temp] Add frontend ingress * [main.yml] Fix CKAN_IMAGE for building * [main.yml] Fix Trivy repo variable * Add Site Title for frontend env * Enable auth and fix frontend svc * Run builds on dev and PRs to dev only * Changing text color * Fix cost splitting CI * [images][l]: add images * [package][s]: add headlessui * [taildwind.config][m]: add custom themes * [components][l]: implement shared and locallize components * [footer][m]: implement footer component * [Header][s]: implement header component * [_documents][s]: add font url to document level * [Search][m]: implement search page * [search-page][s]: remove div and use fragment * [fix][l]: update styling * [package][s]: add swiper * [Carousel][m]: add swiper carousel * [carouselnavbutton][m]: add carousel nav button * [Carousel][l]: wrap higlight and recetly sections inide carousel * [Footer][s]: fix footer alignment issue * [Header][s]: remove comment * [Carousel][s]: remove state * fix spacing to correct vieport * Shell for search page * Advanced search * Fix build * feat(odp-136): implement explore topics page UI * fix(odp-136): eslint errors * Little fixes * Text-xs on small mobile * Individual page for topic * Copy button * Acumin Pro used * Add WRI extension and update CKAN package schema (#20) * Sync dev with master (#19) * Fix/broken build (#13) * [cost-splitting-report.yml] Add permissions * [values.yaml.dev.template] Add quotes to the bool env variables --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> * Update cost-splitting-report.yml --------- Co-authored-by: Luccas Mateus * Add wri extension and update package schema * Add gitkeep for src directory * Update gitignore * Use symlink for ckanext-wri * Fix tests * [GitHub Actions] try ckan-dev instead of localhost * [GitHub Actions] Add wait-on to docker step * [GitHub Actions] Revert wait-on * [GitHub Actions] Fix repo path * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension - fix path * [GitHub Actions] Install wri extension - fix path and add APP_DIR * [GitHub Actions] Install wri extension - try symlink in cwd * [GitHub Actions] Install wri extension - remove symlink and update path * [GitHub Actions] Install wri extension - add debug log * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension * [GitHub Actions] Install wri extension - mv extension during action * [GitHub Actions] Install wri extension - mv extension during action * [GitHub Actions] Install wri extension - mv extension during action * [GitHub Actions] Fix tests * [GitHub Actions] Generate JWT tokens * [GitHub Actions] Install openssl * [GitHub Actions] Install openssl as root * [GitHub Actions] Fix key generation * [GitHub Actions] Fix key generation - remove pre-generated keys * [GitHub Actions] Test adding plugins directly in test * [GitHub Actions] Update scheming env variables * Fix schema validators * Fix schema validators * Remove debugging output * Dummy commit to run CI/CD * Update docs --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> Co-authored-by: Luccas Mateus * fix: minor typo on classname * [Odp 147]:Team metadata page implementation [mockup implementation] (#35) * [tailwindconfig][s]" add maxwidth * add icon and image * Team page implementation * [TeamHeader][l]: update styling * group component * add activity icons * [ActivityStream][l]: add activity stream components * [DatasetList][l]: component to list all dataset * [Teams][xl]: add teams page * update teams page- move to a subpage * update vertical space * Little fixes --------- Co-authored-by: Luccas Mateus * build: trigger reubild * feat(odp-137): implement UI for explore teams page * Enable scheming extension on test site * build: dummy commit to trigger build * Dummy commit to rerun CI/CD * Another dummy commit to rerun CI/CD * Another dummy commit to rerun CI/CD * Trigger CI * build: dummy commit to trigger build * Temporarily remove unit tests * Fix schema tests * Fixes odp-134 - Misc (#39) * fixe alignment issues * [Recent][s]:add shadows to bottom * update styling * Fix responsiveness * Fix icons * [TeamSearchResults][[s]: add pagination * fix: active link effect and login button spacing on mobile (#41) * fix: active link effect and login button spacing on mobile * Change route name to match url --------- Co-authored-by: Luccas Mateus * Fix share topic button (#43) * [values.yaml.dev.template] Increase frontend resources (#44) * fix styling and spacing (#45) * [main.yml] Fix frontend build secrets * [cost-splitting] Change the links according to the branch * [cost-splitting] Update branch name * Avoid overwriting cost splitting files of staging * Fix url for cypress tests --------- Co-authored-by: GitHub Action Co-authored-by: Luccas Mateus Co-authored-by: Michael Polidori Co-authored-by: steveoni Co-authored-by: João Demenech Co-authored-by: Demenech * [main.yml] Use same url for int tests as dev * [codeql] Use staging branch * Sync staging to dev (#105) * Fix Staging Title * Fix download link (#136) (#137) * Fix download link * Change update to patch * [Bypassed][Dockerfile] Disable public signup * Remove test.yml as not currently needed * Merge Dev to Staging (#201) * fix(map view): prevent client error when legend config is empty * Feature/pending datasets (#198) * Add pending_datasets table init and CRUD * Add approval_status and draft to metadata * Fix draft permissions * [main.yml] Init. pendingdb tables * Run WRI unit tests first * Fix paths in unit tests script * Run WRI unit tests first in src * Fix OR in unit tests script --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> * Rm test.yml (#202) --------- Co-authored-by: Demenech Co-authored-by: João Demenech Co-authored-by: Michael Polidori Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> * Datapusher docs * Update docs * Fix typo * Add retry * Fix test * Fix tests * Fix datapusher * Fix types * Revert "Merge branch 'staging' of https://github.com/wri/wri-odp into docs-datapusher-tabular-preview" This reverts commit 2d72215fcb9229ba76e64f6698c4bdcb54e3b987, reversing changes made to b5b90fa27c13619c7068572207c9057df69c12e9. --------- Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> Co-authored-by: MuhammadIsmailShahzad Co-authored-by: GitHub Action Co-authored-by: Michael Polidori Co-authored-by: steveoni Co-authored-by: João Demenech Co-authored-by: Demenech * update activity stream to fetch filter data via server request (#241) * Make datapusher not conflict with multiple environmetns (#243) * Make datapusher not conflict with multiple environmetns * fix typo * Trigger CI (#244) * Trigger CI * Trigger CI * Add retries * Fix: Create resource view for non-public datasets not working (#245) * fix(chart view): auth header not beingproperly passed * trigger build * Feat/api endpoints (#246) * feat: add API endpoints information to dataset and datafile metadata sections * feat(docs): add API endpoints docs * feat(tests): add a test case for API endpoints * fix: minor issue with syntax highlighting * fix: missing dependency * fix(API endpoints): prevent double slashes on endpoints URLs (#248) * Decoded layers start (#249) * approval workflow createform (#224) * approval workflow createform * update approval create form * update on edit andtabular resources * edit datasete * update approval workflow: create and edit form * fix test * fix test * fix type and data visibility * fix test * update documentation * update test * update test * update test * fix test * update test * update test * fix test * increase timeout * fix test * Odp 197 (#250) * Progress * Progress * ODP-197 * Fix datapusher test * Fix build * Fix tests * Fix tests * fix wrong table fields (#251) * Manage how data enters approval state (#253) * fix wrong table fields * filter out updated value in editform * filter updated fields * Documentation ODP-197 (#252) * Documentation ODP-197 * Add animation * Feat/file conversion (#254) * feat(dataset page): refactor datafile download button * feat(dataset page): add datafile download dropdown * feat: converted data files download * feat(downloads): add docs * fix: update envvars for dev/tests envs * fix lint * Dummy commit * fix(file download): fix expiration time for signed URL + missing key in download_callback (#255) * disable notification for private and draft (#256) * Decode functions second try (#257) * QA Fixes (#258) * QA Fixes * Trigger CI * Fix dataset creation (#259) * Make tests more robust (#260) * Fix dataset creation * Make tests more robus * Fix/file conversion qa (#261) * fix(file download): make layers use the layer SQL * feat(views): tweak add view button labels * fix(file download): make filenames human-readable * feat(views): create chart button now shows a tooltip when chart is not creatable i.e. data file is not pushed to DataStore * fix(file download): sanitize headers when converting to XML * fix(file download): fix ODP link in the download file email template * fix(views): fix test that checks whether the View Chart Preview button is present * fix(views): fix test that checks whether the View Table Preview button is present * fix(file download): fix SQL query for Layers * fix(file download): add simple test * Change text (#262) * fixes based on qa (#263) * disable notification for private and draft * fix chart view test * update based on qa * fix type error * fix test * increase timeout * fix reject approval * fix test * increase timeout * comment out banner check * QA fixes - Update views and data files (#265) * fix(resource edit): fix syntax error * fix(view edit): redirect after save not working when editing layers + remove console logs * fix(dataset create test): commented out line that checks notification after editing dataset because of redirect * fix: increate retries for chart view test * manage approval dataset display and qa fixes (#266) * manage approval public and fix qa * update pensing data display * fix test * update workflow * fix build * Decoded functions 3 (#264) * Change text * Decoded functions 3 * Fix tooltip interaction form (#267) * fix approval issues (#268) * fix approval issues * add sysadmin check * fix topics not displaying * fix test * Fix client errro feb 12 (#269) * GLAD Alerts (#273) * feat(API endpoints): implement code snippets for Python and R (#272) * fix(file download): add authorization header when requesting to download a file from a private dataset (#275) * fix page content not loading on navigation [ODP-199] (#274) * fix page content not loading on navigation * moved key from global to local scope of teams and topics * Approval workflow for private datasets (#277) * Approval workflow for private datasets * Trigger CI * Trigger CI * Fixes/fe improvements (#278) * fix(maps): icon for hide/show layer not changing according to layer state * fix(maps): legends toggle arrow alignment * feat(sign in): redirect to dashbord after sign in * fix(sign in): when signing in with azure, show loading indicator on the correct button * trigger build * Fix inconsistent number of dataset under topics and teams (#279) * fix page content not loading on navigation * get the right number of datset and count * Edited alerts (background change) (#280) * Fix licenses (#281) * Fix datafiles highlight (#283) * Fix null user on approvalrow component (#282) * Prettier diff (#285) * Prettier diff * Trigger CI * Replace resources/organization with datafiles/teams * Fix tests * Try to fix notifications (#287) * Try to fix notifications * Fix build * Change title * Fix dataset pushed to pending when collaborator is added (#288) * fix page content not loading on navigation * fix collaborator fields not saved alone * [Odp 203]: Default data preview (#286) * fix page content not loading on navigation * add default view * fix types * update default view * fix double rendring of smae layer * fix dataset version * remove preview tab * Only show featured for public datasets (#289) * Only show featured for public datasets * only show featured when public is selected * feat(release notes): implementation + docs + tests (#291) * Fix changing resource title name (#290) * Fix logan (#292) * Fix logan * Fix resources not being created * Seo - [ODP-23] (#293) * fix page content not loading on navigation * add seo tags * fix invlaid config * add next-sitemap * add seo config * Fix datapreview logic (#295) * fix page content not loading on navigation * fix data preview logic * Release Notes QA Fixes (#297) * fix(release notes): make release notes field RTE + fix scrollbars * fix(tests): update tests to work with new release notes editor * fix(tests): update tests to work with new release notes editor * fix(tests): update tests to work with new release notes editor * display tags and topics (#296) * display tags and topics * Trigger CI * Fix tests * Fix tests * Fix tests * Fix unit tests * Fix tabular preview taking too long to load, due to waiting for session apikey to get resolve on initial load (#299) * fix page content not loading on navigation * fix preview wating for session to get resolve * fix(release notes): display release notes in descending order (#301) * Fix dataset being just created (#302) Co-authored-by: Luccas Mateus * Disable datapusher for links (#300) * Disable datapusher for links * Fix typo * Fix typo * Show datasettitle on toaster * Use url_type instead of type * Fix types * Trigger CI * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix tests * Fix conversion test * More domains * Fix usernames in notification emails * Fix bread crumb (#306) * fix(release notes): fix preview when creating dataset + fix order (#308) * fix(release notes): fix preview when creating dataset + fix order * fix(release notes): fix release notes rendering wrong on approval row * Fix/release notes general (#309) * fix(release notes): fix preview when creating dataset + fix order * fix(release notes): fix release notes rendering wrong on approval row * fix(release notes): fix release notes not showing for newly created datasets + minor rendering issue --------- Co-authored-by: steveoni Co-authored-by: Michael Polidori Co-authored-by: Muhammad Ismail Shahzad <57398621+MuhammadIsmailShahzad@users.noreply.github.com> Co-authored-by: Luccas Mateus Co-authored-by: MuhammadIsmailShahzad Co-authored-by: GitHub Action --- .github/workflows/main.yml | 3 +- ckan-backend-dev/.env.example | 1 + ckan-backend-dev/docker-compose.test.yml | 1 + .../ckanext/wri/logic/action/get.py | 18 ++-- .../src/ckanext-wri/ckanext/wri/plugin.py | 1 + .../wri/tests/test_pending_datasets.py | 5 +- deployment/frontend/Dockerfile | 1 + deployment/frontend/next-sitemap.config.js | 22 +++++ deployment/frontend/next.config.mjs | 58 +++++++------ deployment/frontend/package-lock.json | 58 +++++++++++++ deployment/frontend/package.json | 6 +- deployment/frontend/public/robots.txt | 7 ++ .../dashboard/approval/ApprovalRow.tsx | 61 ++++++++++++- .../dashboard/datasets/DatasetRow.tsx | 12 ++- .../datasets/admin/datafiles/EditDataFile.tsx | 4 +- .../datasets/admin/metadata/RTE/Icons.tsx | 2 + .../admin/metadata/RTE/SimpleEditor.tsx | 21 +++++ .../admin/metadata/VersioningForm.tsx | 9 +- .../datasets/admin/preview/Preview.tsx | 20 ++++- .../components/data-explorer/queryHooks.ts | 6 +- .../datasets/ApprovalRequestCard.tsx | 6 +- .../datasets/sections/Versioning.tsx | 68 +++++++++------ .../visualizations/Visualizations.tsx | 1 + deployment/frontend/src/env.mjs | 2 + deployment/frontend/src/pages/_app.tsx | 20 ++++- .../src/pages/dashboard/activity-stream.tsx | 37 +++++--- .../src/pages/dashboard/approval-request.tsx | 37 +++++--- .../src/pages/dashboard/datasets/index.tsx | 39 +++++---- .../frontend/src/pages/dashboard/index.tsx | 85 ++++++++++-------- .../src/pages/dashboard/notifications.tsx | 11 ++- .../src/pages/dashboard/teams/index.tsx | 39 +++++---- .../src/pages/dashboard/topics/index.tsx | 38 ++++---- .../frontend/src/pages/dashboard/users.tsx | 38 ++++---- .../src/pages/dashboard/users/index.tsx | 11 +++ .../pages/datasets/[datasetName]/index.tsx | 87 +++++++++++-------- deployment/frontend/src/pages/index.tsx | 13 ++- deployment/frontend/src/pages/search.tsx | 11 ++- .../frontend/src/pages/search_advanced.tsx | 13 ++- .../src/pages/server-sitemap.xml/index.tsx | 15 ++++ .../frontend/src/pages/teams/[teamsName].tsx | 16 +++- deployment/frontend/src/pages/teams/index.tsx | 12 ++- .../frontend/src/pages/topics/[topicName].tsx | 12 ++- .../frontend/src/pages/topics/index.tsx | 12 ++- .../frontend/src/schema/dataset.schema.ts | 1 + .../src/server/api/routers/dataset.ts | 45 ++++++---- deployment/frontend/src/styles/rte.css | 11 ++- deployment/frontend/src/utils/apiUtils.ts | 51 ++++++++++- deployment/frontend/src/utils/general.ts | 2 +- e2e-tests/cypress/e2e/chart_view.cy.js | 43 ++++----- .../cypress/e2e/dataset_create_and_read.cy.js | 12 +-- e2e-tests/cypress/e2e/file-conversion.cy.js | 40 +++++---- e2e-tests/cypress/e2e/release_notes.cy.js | 4 +- e2e-tests/cypress/fixtures/airtravel.csv | 14 +++ 53 files changed, 848 insertions(+), 314 deletions(-) create mode 100644 deployment/frontend/next-sitemap.config.js create mode 100644 deployment/frontend/public/robots.txt create mode 100644 deployment/frontend/src/pages/server-sitemap.xml/index.tsx create mode 100644 e2e-tests/cypress/fixtures/airtravel.csv diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 2cbc3b40e..b7dd44c97 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -67,6 +67,7 @@ jobs: --build-arg NEXTAUTH_URL=${{ secrets.STAGING_FRONTEND_NEXTAUTH_URL }} \ --build-arg CKAN_URL=${{ secrets.STAGING_FRONTEND_CKAN_URL }} \ --build-arg NEXT_PUBLIC_CKAN_URL=${{ secrets.STAGING_FRONTEND_CKAN_URL }} \ + --build-arg NEXT_PUBLIC_NEXTAUTH_URL=${{ secrets.STAGING_FRONTEND_NEXTAUTH_URL }} \ deployment/frontend docker push $REGISTRY/$FRONTEND_REPO:$IMAGE_TAG - name: Build and push Datapusher image to ECR @@ -248,4 +249,4 @@ jobs: - name: 'Deploy using Helm Upgrade' run: | set -e - helm upgrade -i dx-helm-wri-$BRANCH_NAME-release ./deployment/helm-templates -f ./deployment/helm-templates/values.yaml -n $PROJECT_NAME-$BRANCH_NAME --create-namespace --wait \ No newline at end of file + helm upgrade -i dx-helm-wri-$BRANCH_NAME-release ./deployment/helm-templates -f ./deployment/helm-templates/values.yaml -n $PROJECT_NAME-$BRANCH_NAME --create-namespace --wait diff --git a/ckan-backend-dev/.env.example b/ckan-backend-dev/.env.example index d58a89b1f..7a57a8e2d 100644 --- a/ckan-backend-dev/.env.example +++ b/ckan-backend-dev/.env.example @@ -128,6 +128,7 @@ NEXTAUTH_SECRET=secret NEXTAUTH_URL=http://frontend:3000 CKAN_URL=http://ckan-dev:5000 NEXT_PUBLIC_CKAN_URL=http://ckan-dev:5000 +NEXT_PUBLIC_NEXTAUTH_URL=http://frontend:3000 S3_ACCESS_KEY_ID=${MINIO_ROOT_USER} S3_SECRET_KEY_ID=${MINIO_ROOT_PASSWORD} S3_BUCKET_NAME=ckan diff --git a/ckan-backend-dev/docker-compose.test.yml b/ckan-backend-dev/docker-compose.test.yml index 0a7c2bdc2..c4292914e 100755 --- a/ckan-backend-dev/docker-compose.test.yml +++ b/ckan-backend-dev/docker-compose.test.yml @@ -42,6 +42,7 @@ services: context: ../deployment/frontend/ args: - NEXT_PUBLIC_CKAN_URL=http://ckan-dev:5000 + - NEXT_PUBLIC_NEXTAUTH_URL=http://localhost:3000 environment: - NEXTAUTH_SECRET=secret - NEXTAUTH_URL=http://localhost:3000 diff --git a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py index 8daedec8c..d79c42f85 100644 --- a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py +++ b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py @@ -481,6 +481,7 @@ def pending_diff_show(context: Context, data_dict: DataDict): dataset_diff = None + pending_dataset = None try: pending_dataset = PendingDatasets.get(package_id=package_id) log.error("===============pending dataset=================") @@ -499,7 +500,11 @@ def pending_diff_show(context: Context, data_dict: DataDict): # _("Diff not found for Pending Dataset: {}".format(package_id)) # ) - return dataset_diff + return { + "diff": dataset_diff, + "old_dataset": existing_dataset, + "new_dataset": pending_dataset, + } def _diff(existing, pending, path=""): @@ -542,13 +547,15 @@ def _process_lists(existing_list, pending_list, path): return list_diff + @logic.side_effect_free def dataset_release_notes(context: Context, data_dict: DataDict): model = context["model"] conn = model.Session.connection() id = data_dict.get("id") - sql = text(''' + sql = text( + """ select distinct on ((data::json->>'package')::json->>'release_notes') ((data::json->>'package')::json->>'release_notes') as release_notes, @@ -561,11 +568,12 @@ def dataset_release_notes(context: Context, data_dict: DataDict): and ((data::json->>'package')::json->>'release_notes') != '' order by 1, 2 DESC - '''.format(id)) + """.format( + id + ) + ) q = conn.execute(sql) results = q.all() return results - - diff --git a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/plugin.py b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/plugin.py index d478063a0..3000f8232 100644 --- a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/plugin.py +++ b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/plugin.py @@ -210,6 +210,7 @@ def _submit_to_datapusher(self, resource_dict: dict[str, Any]): resource_format and resource_format.lower() in supported_formats and resource_dict.get('url_type') != u'datapusher' + and resource_dict.get('url_type') == u'upload' ) if not submit: diff --git a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/tests/test_pending_datasets.py b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/tests/test_pending_datasets.py index 2f12ff725..c3264eeb5 100644 --- a/ckan-backend-dev/src/ckanext-wri/ckanext/wri/tests/test_pending_datasets.py +++ b/ckan-backend-dev/src/ckanext-wri/ckanext/wri/tests/test_pending_datasets.py @@ -114,10 +114,11 @@ def test_pending_diff_show(): get_action("pending_dataset_update")(context, data_dict) - result = get_action("pending_diff_show")(context, {"package_id": dataset["id"]}) + _result = get_action("pending_diff_show")(context, {"package_id": dataset["id"]}) + result = _result['diff'] assert result["title"]["new_value"] == "New Title" assert result["notes"]["new_value"] == "New description" assert result["private"]["new_value"] is True assert result["rw_dataset"]["new_value"] is False - assert result["wri_data"]["new_value"] is True \ No newline at end of file + assert result["wri_data"]["new_value"] is True diff --git a/deployment/frontend/Dockerfile b/deployment/frontend/Dockerfile index 87885f983..f4e1b21a3 100644 --- a/deployment/frontend/Dockerfile +++ b/deployment/frontend/Dockerfile @@ -11,6 +11,7 @@ ARG NEXTAUTH_SECRET ARG NEXTAUTH_URL ARG CKAN_URL ARG NEXT_PUBLIC_CKAN_URL +ARG NEXT_PUBLIC_NEXTAUTH_URL ARG AZURE_AD_CLIENT_ID ARG AZURE_AD_CLIENT_SECRET ARG AZURE_AD_TENANT_ID diff --git a/deployment/frontend/next-sitemap.config.js b/deployment/frontend/next-sitemap.config.js new file mode 100644 index 000000000..8ef01af88 --- /dev/null +++ b/deployment/frontend/next-sitemap.config.js @@ -0,0 +1,22 @@ +console.log('[*] SITE_URL', process.env.NEXT_PUBLIC_NEXTAUTH_URL) + +module.exports = { + siteUrl: process.env.NEXT_PUBLIC_NEXTAUTH_URL, + generateRobotsTxt: true, + exclude: ['/server-sitemap.xml', '/dashboard/*', '/dashboard'], + robotsTxtOptions: { + additionalSitemaps: [ + `${process.env.NEXT_PUBLIC_NEXTAUTH_URL}/server-sitemap.xml`, + ], + policies: [ + { + userAgent: '*', + disallow: ['/dashboard/*'] + }, + { + userAgent: '*', + allow: '/', + } + ], + } +}; \ No newline at end of file diff --git a/deployment/frontend/next.config.mjs b/deployment/frontend/next.config.mjs index b360b2141..aabc3efd2 100644 --- a/deployment/frontend/next.config.mjs +++ b/deployment/frontend/next.config.mjs @@ -3,37 +3,45 @@ * for Docker builds. */ -import withBundleAnalyzer from '@next/bundle-analyzer'; +import withBundleAnalyzer from '@next/bundle-analyzer' -await import("./src/env.mjs"); +await import('./src/env.mjs') const bundleAnalyzer = withBundleAnalyzer({ - enabled: process.env.ANALYZE === 'true', + enabled: process.env.ANALYZE === 'true', }) /** @type {import("next").NextConfig} */ const config = { - eslint: { - // Warning: This allows production builds to successfully complete even if - // your project has ESLint errors. - ignoreDuringBuilds: true, - }, - reactStrictMode: true, - output: "standalone", + eslint: { + // Warning: This allows production builds to successfully complete even if + // your project has ESLint errors. + ignoreDuringBuilds: true, + }, + reactStrictMode: true, + output: 'standalone', + /** + * If you are using `appDir` then you must comment the below `i18n` config out. + * + * @see https://github.com/vercel/next.js/issues/41980 + */ + i18n: { + locales: ['en'], + defaultLocale: 'en', + }, + images: { + domains: [ + 'wri-dev-ckan-svc', + 'wri.dev.ckan.datopian.com', + 'wri.staging.ckan.datopian.com', + 'wri.prod.ckan.datopian.com', + 'test-bucket-wri.s3.ap-northeast-1.amazonaws.com', + 'ckan-dev', + 'gravatar.com', + 's3.amazonaws.com', + ], + }, +} - /** - * If you are using `appDir` then you must comment the below `i18n` config out. - * - * @see https://github.com/vercel/next.js/issues/41980 - */ - i18n: { - locales: ["en"], - defaultLocale: "en", - }, - images: { - domains: ['wri-dev-ckan-svc', 'wri.dev.ckan.datopian.com', 'test-bucket-wri.s3.ap-northeast-1.amazonaws.com', 'ckan-dev', 'gravatar.com', 's3.amazonaws.com'], - }, -}; - -export default bundleAnalyzer(config); +export default bundleAnalyzer(config) diff --git a/deployment/frontend/package-lock.json b/deployment/frontend/package-lock.json index fb16cb6cc..6d917e639 100644 --- a/deployment/frontend/package-lock.json +++ b/deployment/frontend/package-lock.json @@ -38,11 +38,13 @@ "@tanstack/react-query": "^4.32.6", "@tanstack/react-table": "^8.10.7", "@tiptap/extension-bold": "^2.1.12", + "@tiptap/extension-bullet-list": "^2.2.3", "@tiptap/extension-code": "^2.1.12", "@tiptap/extension-document": "^2.1.12", "@tiptap/extension-history": "^2.1.12", "@tiptap/extension-italic": "^2.1.12", "@tiptap/extension-link": "^2.1.12", + "@tiptap/extension-list-item": "^2.2.3", "@tiptap/extension-paragraph": "^2.1.12", "@tiptap/extension-placeholder": "^2.1.13", "@tiptap/extension-strike": "^2.1.12", @@ -80,6 +82,7 @@ "next": "^13.5.4", "next-auth": "^4.24.5", "next-seo": "^6.1.0", + "next-sitemap": "^4.2.3", "nodemailer": "^6.7.8", "old-vizzuality-components": "npm:vizzuality-components@1.3.0", "papaparse": "^5.4.1", @@ -1657,6 +1660,11 @@ "resolved": "https://registry.npmjs.org/commander/-/commander-2.20.3.tgz", "integrity": "sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==" }, + "node_modules/@corex/deepmerge": { + "version": "4.0.43", + "resolved": "https://registry.npmjs.org/@corex/deepmerge/-/deepmerge-4.0.43.tgz", + "integrity": "sha512-N8uEMrMPL0cu/bdboEWpQYb/0i2K5Qn8eCsxzOmxSggJbbQte7ljMRoXm917AbntqTGOzdTu+vP3KOOzoC70HQ==" + }, "node_modules/@danmarshall/deckgl-typings": { "version": "4.9.28", "resolved": "https://registry.npmjs.org/@danmarshall/deckgl-typings/-/deckgl-typings-4.9.28.tgz", @@ -5889,6 +5897,18 @@ "@tiptap/pm": "^2.0.0" } }, + "node_modules/@tiptap/extension-bullet-list": { + "version": "2.2.3", + "resolved": "https://registry.npmjs.org/@tiptap/extension-bullet-list/-/extension-bullet-list-2.2.3.tgz", + "integrity": "sha512-BpYg1pIfLE+2LTC90ts53deEWGSmAojhM/jJ84U19qfbfXt/7/KHrZJ4SAMxJSW3pLpy0bIq2XuOuvppOYVR5g==", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/ueberdosis" + }, + "peerDependencies": { + "@tiptap/core": "^2.0.0" + } + }, "node_modules/@tiptap/extension-code": { "version": "2.1.12", "resolved": "https://registry.npmjs.org/@tiptap/extension-code/-/extension-code-2.1.12.tgz", @@ -5970,6 +5990,18 @@ "@tiptap/pm": "^2.0.0" } }, + "node_modules/@tiptap/extension-list-item": { + "version": "2.2.3", + "resolved": "https://registry.npmjs.org/@tiptap/extension-list-item/-/extension-list-item-2.2.3.tgz", + "integrity": "sha512-eyfk4f1jOioj+mkIN2m6XQK61MpV0fi17utt8VNx893Td8kS0g7HHuuYMwyjIRtG35ENUaAt7c216JQwnLsrAw==", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/ueberdosis" + }, + "peerDependencies": { + "@tiptap/core": "^2.0.0" + } + }, "node_modules/@tiptap/extension-paragraph": { "version": "2.1.12", "resolved": "https://registry.npmjs.org/@tiptap/extension-paragraph/-/extension-paragraph-2.1.12.tgz", @@ -15248,6 +15280,32 @@ "react-dom": ">=16.0.0" } }, + "node_modules/next-sitemap": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/next-sitemap/-/next-sitemap-4.2.3.tgz", + "integrity": "sha512-vjdCxeDuWDzldhCnyFCQipw5bfpl4HmZA7uoo3GAaYGjGgfL4Cxb1CiztPuWGmS+auYs7/8OekRS8C2cjdAsjQ==", + "funding": [ + { + "url": "https://github.com/iamvishnusankar/next-sitemap.git" + } + ], + "dependencies": { + "@corex/deepmerge": "^4.0.43", + "@next/env": "^13.4.3", + "fast-glob": "^3.2.12", + "minimist": "^1.2.8" + }, + "bin": { + "next-sitemap": "bin/next-sitemap.mjs", + "next-sitemap-cjs": "bin/next-sitemap.cjs" + }, + "engines": { + "node": ">=14.18" + }, + "peerDependencies": { + "next": "*" + } + }, "node_modules/next-tick": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/next-tick/-/next-tick-1.1.0.tgz", diff --git a/deployment/frontend/package.json b/deployment/frontend/package.json index de031bc5a..db64ff787 100644 --- a/deployment/frontend/package.json +++ b/deployment/frontend/package.json @@ -8,7 +8,8 @@ "lint": "next lint", "start": "next start", "test": "vitest", - "test-ci": "vitest run" + "test-ci": "vitest run", + "postbuild": "next-sitemap" }, "dependencies": { "@aws-sdk/client-s3": "^3.441.0", @@ -41,11 +42,13 @@ "@tanstack/react-query": "^4.32.6", "@tanstack/react-table": "^8.10.7", "@tiptap/extension-bold": "^2.1.12", + "@tiptap/extension-bullet-list": "^2.2.3", "@tiptap/extension-code": "^2.1.12", "@tiptap/extension-document": "^2.1.12", "@tiptap/extension-history": "^2.1.12", "@tiptap/extension-italic": "^2.1.12", "@tiptap/extension-link": "^2.1.12", + "@tiptap/extension-list-item": "^2.2.3", "@tiptap/extension-paragraph": "^2.1.12", "@tiptap/extension-placeholder": "^2.1.13", "@tiptap/extension-strike": "^2.1.12", @@ -83,6 +86,7 @@ "next": "^13.5.4", "next-auth": "^4.24.5", "next-seo": "^6.1.0", + "next-sitemap": "^4.2.3", "nodemailer": "^6.7.8", "old-vizzuality-components": "npm:vizzuality-components@1.3.0", "papaparse": "^5.4.1", diff --git a/deployment/frontend/public/robots.txt b/deployment/frontend/public/robots.txt new file mode 100644 index 000000000..86b39722b --- /dev/null +++ b/deployment/frontend/public/robots.txt @@ -0,0 +1,7 @@ +# Block all crawlers for /dashboard +User-agent: * +Disallow: /dashboard + +# Allow all crawlers +User-agent: * +Allow: / \ No newline at end of file diff --git a/deployment/frontend/src/components/dashboard/approval/ApprovalRow.tsx b/deployment/frontend/src/components/dashboard/approval/ApprovalRow.tsx index 2e700aeb0..f13f002f0 100644 --- a/deployment/frontend/src/components/dashboard/approval/ApprovalRow.tsx +++ b/deployment/frontend/src/components/dashboard/approval/ApprovalRow.tsx @@ -85,6 +85,7 @@ function filteredDataset(dataset: WriDataset) { { title: 'Release Notes', description: dataset?.release_notes ?? '', + isHtml: true }, ] } @@ -174,6 +175,7 @@ function FormatNewValue( return
}) .with(null, () => Not specified) + .with(P.array(P.nullish), () => Empty) .with(P.number, (value) => {value}) .with(P.array(P.string), (value) => { if ( @@ -238,11 +240,52 @@ function SubCardProfile({ data, }: { isLoading: boolean - diff: Record + diff: { + diff: Record + old_dataset: WriDataset | null + new_dataset: WriDataset | null + } data: WriDataset }) { if (isLoading) return - const diff2 = formatDiff(diff) + let diff2 = formatDiff(diff ? diff.diff : null) + // if the diff has tags or topics, instead of showing the diff we show the whole old and new values + if (diff2 && Object.keys(diff2).some((x) => x.includes('tags'))) { + diff2 = Object.fromEntries( + Object.entries(diff2).filter(([key]) => !key.includes('tags')) + ) + diff2 = { + ...diff2, + Tags: { + old_value: + diff.old_dataset?.tags?.map( + (t) => t.display_name ?? t.name + ) ?? [], + new_value: + diff.new_dataset?.tags?.map( + (t) => t.display_name ?? t.name + ) ?? [], + }, + } + } + if (diff2 && Object.keys(diff2).some((x) => x.includes('Topics'))) { + diff2 = Object.fromEntries( + Object.entries(diff2).filter(([key]) => !key.includes('Topics')) + ) + diff2 = { + ...diff2, + Topics: { + old_value: + diff.old_dataset?.groups?.map( + (t) => t.display_name ?? t.name + ) ?? [], + new_value: + diff.new_dataset?.groups?.map( + (t) => t.display_name ?? t.name + ) ?? [], + }, + } + } return (
{diff && @@ -268,6 +311,7 @@ function SubCardProfile({ {Object.keys(diff).filter( (x) => !matchesAnyPattern(x) ).length > 0 && + diff2 && Object.keys(diff2).map((key) => { return ( - {key.description} + {!key.isHtml ? ( + key.description + ) : ( +
+ )} ) diff --git a/deployment/frontend/src/components/dashboard/datasets/DatasetRow.tsx b/deployment/frontend/src/components/dashboard/datasets/DatasetRow.tsx index 99123f122..3f0fa7f7c 100644 --- a/deployment/frontend/src/components/dashboard/datasets/DatasetRow.tsx +++ b/deployment/frontend/src/components/dashboard/datasets/DatasetRow.tsx @@ -38,6 +38,7 @@ function subFields(dataset: WriDataset) { { title: 'Release Notes', description: dataset?.release_notes, + isHtml: true, }, ] } @@ -106,7 +107,16 @@ function SubCardProfile({ dataset }: { dataset: WriDataset }) { {item.title}

- {item.description} + {!item.isHtml ? ( + item.description + ) : ( +

+ )}

) diff --git a/deployment/frontend/src/components/dashboard/datasets/admin/datafiles/EditDataFile.tsx b/deployment/frontend/src/components/dashboard/datasets/admin/datafiles/EditDataFile.tsx index 4d86dfe70..32bbbacaa 100644 --- a/deployment/frontend/src/components/dashboard/datasets/admin/datafiles/EditDataFile.tsx +++ b/deployment/frontend/src/components/dashboard/datasets/admin/datafiles/EditDataFile.tsx @@ -254,7 +254,7 @@ export function EditDataFile({ ].includes( datafile.format?.toLowerCase() ?? 'none' - ) && ( + ) && datafile.url_type === 'upload' && ( {({ selected }) => (
diff --git a/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/Icons.tsx b/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/Icons.tsx index aa891b613..523a67da1 100644 --- a/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/Icons.tsx +++ b/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/Icons.tsx @@ -1,3 +1,4 @@ +import { ListBulletIcon } from "@heroicons/react/24/outline"; import React from "react"; export const Bold = ({ size = 16, color = "currentColor" }) => ( @@ -149,3 +150,4 @@ export const X = ({ size = 16, color = "currentColor" }) => ( ); +export const ListItem = ListBulletIcon diff --git a/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/SimpleEditor.tsx b/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/SimpleEditor.tsx index a3be31c8d..3db9a2832 100644 --- a/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/SimpleEditor.tsx +++ b/deployment/frontend/src/components/dashboard/datasets/admin/metadata/RTE/SimpleEditor.tsx @@ -11,6 +11,8 @@ import Underline from '@tiptap/extension-underline' import Italic from '@tiptap/extension-italic' import Strike from '@tiptap/extension-strike' import Code from '@tiptap/extension-code' +import BulletList from '@tiptap/extension-bullet-list' +import ListItem from '@tiptap/extension-list-item' import History from '@tiptap/extension-history' // Custom import * as Icons from './Icons' @@ -98,6 +100,12 @@ function TipTapEditor({ // Use a placeholder: placeholder: placeholder ?? '', }), + BulletList, + ListItem.configure({ + HTMLAttributes: { + class: 'list-disc', + }, + }), ], }) as Editor @@ -153,6 +161,10 @@ function TipTapEditor({ editor.chain().focus().toggleCode().run() }, [editor]) + const toggleList = useCallback(() => { + editor.chain().focus().toggleBulletList().run() + }, [editor]) + if (!editor) { return null } @@ -235,6 +247,15 @@ function TipTapEditor({ > +
-