Feat/8 handle multiple databases and users #20

pmayd · 2023-10-27T13:36:09Z

but open for comments

closes #8

…antly reduced lines of code by getting rid of the settings.ini

…onfig. Can set the current active database and get the settings from the config

…pported versions to 3.10 and 3.11

pmayd · 2023-10-29T20:29:53Z

PR can be reviewed work is done

pmayd · 2023-10-29T20:44:25Z

Will fix code quality later

MarcoHuebner

Mostly looked at the notebooks and left some minor comments. The overall logic is IMO easy to use 👍

MarcoHuebner · 2023-11-01T10:29:00Z

.github/workflows/run-tests.yaml

@@ -24,31 +24,31 @@ jobs:
    strategy:
      fail-fast: false
      matrix:
-        python-version: ["3.9", "3.10"]
+        python-version: ["3.10", "3.11"]
        os: [ubuntu-22.04, macOS-latest, windows-latest]


Should we (at some point) switch to ubuntu-latest to not have to update this manually (and forget about it) and to be consistent across OS's

What do you mean? How does using ubuntu latest help in testing different Python versions?

I just meant ubuntu-22.04 -> ubuntu-latest, since this is the way we do it for the other OS's as well

MarcoHuebner · 2023-11-01T10:30:35Z

nb/00_Setup.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install python-dotenv"


Should we instead put python-dotenv into the standard dependency group in the pyproject.toml?

We can but it is more my personal style to use it it is by no means required...true the notebook uses it which is bad style already basically it should be optional and the developer can decide how to use and call setup credentials. But maybe it is not that bad to add it as dependency I just like to keep the dependencies to the bare minimum

MarcoHuebner · 2023-11-01T10:32:32Z

nb/00_Setup.ipynb

+   "outputs": [],
+   "source": [
+    "# only execute if you want to delete your config file for test purposes\n",
+    "config.delete_config()\n",


You risk people accidentally deleting their set up config when using "Run All" - I think this should either be mentioned in a markdown cell or be a # comment by default

It was until I used it and forgot to comment again XD

MarcoHuebner · 2023-11-01T10:47:33Z

nb/00_Setup.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pprint import pprint\n",


MarcoHuebner · 2023-11-01T10:50:27Z

nb/02_helloworld.ipynb

    "import logging\n",
    "logging.basicConfig(level=logging.INFO)\n",


Could be removed, since it does not provide additional insights (no info logged by called functions)

… dir and always use the default config dir under user home.

…define own config path; pystatis supports only default config path but custom data cache path

pmayd · 2023-11-04T14:50:18Z

@MarcoHuebner complete overhaul, I decided to not support custom config paths any longer as it is too much work and makes the code only more complex.

I fixes all tests, they create a new temporary config and restore the old user config, so tests should NO LONGER affect existing user configs!

MarcoHuebner · 2023-11-04T15:49:00Z

I fixes all tests, they create a new temporary config and restore the old user config, so tests should NO LONGER affect existing user configs!

Alright, whatever works for the user in the end 😄 I also addressed overwriting the config_dir in 7ce62ad ( #4 ), but via mocking the settings directory

…ct instead of overwriting the config variable within the functions using global (bad style according to pylint)

* Bump version to next major version #9 * Revert flake8 to ^3.0 for docstrings #9 * add a notebook that shows how to run init_config * Make dev dependencies optional, update lock and README #9 * Update workflow install --with dev, add matrix poetry version #9 * Fix python and poetry version definition #9 * Fix python and poetry version definition #9 * fix lock file * update dev dependencies and add python-dotenv to dev * improve readme * update readme * Feat/8 handle multiple databases and users (#20) * change config module to handle multiple databases * finalize work on config module to handle multiple databases; significantly reduced lines of code by getting rid of the settings.ini * add a new db module that serves as a layer between the user and the config. Can set the current active database and get the settings from the config * simplify config module * refactor code to implement new config; correct tests * fix all remaining tests * fix all text issues * update notebooks according to latest changes in config * drop support for Python 3.9 due to pipe operator for types and set supported versions to 3.10 and 3.11 * fix problem with config dir creation during setup * fix isort * Improve clear_cache output for full wipe, remove unused import * Address all non global-related pylint issues #20 * because of complexity get rid of the current support of custom config dir and always use the default config dir under user home. * fix all tests; get rid of settings.ini and functionality for user to define own config path; pystatis supports only default config path but custom data cache path * fix all tests; get rid of settings.ini and functionality for user to define own config path; pystatis supports only default config path but custom data cache path * refactor config module to work with a ConfigParser global config object instead of overwriting the config variable within the functions using global (bad style according to pylint) * address pylint issues * fix mypy issues * fix pylint issues --------- Co-authored-by: MarcoHuebner <marco_huebner1@gmx.de> * update README to the latest changes of multi database support * Added lists of all available statistics and tables * Feat/10 update and auto deploy sphinx (#27) * Updated dev-dependencies, added first version of Sphinx documentation, including built html documentation. * Added Logo, updated theme, updated GitHub workflow, fixed docstrings in cache and cube. Hosting on ReadTheDocs has to be done by Owner/ CorrelAid (but can be requested and triggered that way). * Updated urllib3 version, but everything <2.0.0 (deprecating `strict`) should be fine... * Updated poetry as recommended in cachecontrol issue report. * Fixed black formatting, fixed make docs (is now ran by poetry). * Fixed linting issue, updated packages, updated make docs. * Updated ReadMe, added developer sphinx documentation, added custom pre-commit hook and changed to hard-coded version in docs, added built documentation to artifacts, #3 * Add deployment workflow, needs Repo updates * Update depencies for Sphinx documentation #10 * Remove redundant docu information #10 Render parts of the README.md in the respective .rst files * Remove unused mdinclude, fix run-test py version, update pre-commit #10 * Fix dependency group for SPhinx workflow #10 * Fix docstring parameter rendering in Sphinx #10 * Fix image rendering by mimicking folder structure #10 * Add comment on warnings related to ext.napoleon #10 * Rename deploy-docs #10 * Fix black format issue in conf.py #10 * Update deploy key, add deploy trigger comment #10 * Update documentation deploy workflow #10 * Switch to matrix.os definition #10 * Fix pull_request target in deploy workflow #10 * Update poetry.lock #10 * Import package version to Sphinx docu #10 * Manually fix black formatting issue #10 * With auto-deploy working, decrease retention days #10 * Update readme and Sphinx header references #10 * Fix deploy to update files on the remote #10 * fix cube functionality: it seems like structure of QEI header part was changed as well as DQA no longer has information about axis so we assume that the order is preserved (#43) * add jupytext and new nb for presentation * Feat/35 implement regex matching (#44) * Implemented regex matching, initial commit * Added credentials check for cubes and removed all references to set_db() * Implemented regex matching, initial commit * Added credentials check for cubes and removed all references to set_db() * fix tests * refactoring Find and Result class to work with new database detection logic; because find does not use names like Table and Cube, use has to specify the database * fix tests --------- Co-authored-by: Michael Aydinbas <michael.aydinbas@gmail.com> Co-authored-by: Michael Aydinbas <michael.aydinbas@new-work.se> * add presentation nb * remove presentation nb for now * Feat/19 improve readability of the table format (#42) * Reformatting the raw data tables for readability * Adding comments * Applied suggested changes and run code formatting * add tests for Table --------- Co-authored-by: Michael Aydinbas <michael.aydinbas@new-work.se> * prepare Table so it can parse data from three different sources * Added description and examples of Find * implement parse logic for prettify zensus tables * fix pylint issues * edits on Find section * fixing overwritten changes * update presentation nb * add genesis parse code for regio, too, for the moment. * Feat/34 visualization examples (#48) * Add 02_Geo_visualization_example.ipynb * changed '-' to 0 instead of nan --> reproduce Simons result * new case study in visualization notebook, integration to presentation notebook * catch NA-values in read_csv and added Auspraegung_Code to table.py to have the unique region identifiers --------- Co-authored-by: jkrause <jkrause123@users.noreply.github.com> * final presentation nb and shape data; omit file check in pre-commit * fixed typo and beautified plots in presentation.ipynb /.py * add a first workaround for the new Zensus zip content type * fix all tests; separate Find and Results classes into own modules * update dependencies * update README * set version to 0.2 * remove Cubes from package for now; we no longer support cubes until they are requested * fix all tests; fix all relevant nb; * fix pylint issues * fix mypy issues * add documentation key * update changelog --------- Co-authored-by: MarcoHuebner <marco_huebner1@gmx.de> Co-authored-by: Pia <45008571+PiaSchroeder@users.noreply.github.com> Co-authored-by: MarcoHuebner <57489799+MarcoHuebner@users.noreply.github.com> Co-authored-by: zosiaboro <50183305+zosiaboro@users.noreply.github.com> Co-authored-by: Zosia Borowska <zofia.anna.borowska@gmail.com> Co-authored-by: jkrause123 <89632018+jkrause123@users.noreply.github.com> Co-authored-by: jkrause <jkrause123@users.noreply.github.com>

pmayd and others added 9 commits October 27, 2023 13:26

change config module to handle multiple databases

d696245

finalize work on config module to handle multiple databases; signific…

4c06192

…antly reduced lines of code by getting rid of the settings.ini

add a new db module that serves as a layer between the user and the c…

0f13d31

…onfig. Can set the current active database and get the settings from the config

simplify config module

c574d5f

refactor code to implement new config; correct tests

857af8e

fix all remaining tests

8fcddc0

fix all text issues

1f74065

update notebooks according to latest changes in config

4675f6d

drop support for Python 3.9 due to pipe operator for types and set su…

28e8f4c

…pported versions to 3.10 and 3.11

pmayd requested a review from MarcoHuebner October 29, 2023 20:29

pmayd added 2 commits October 29, 2023 21:40

fix problem with config dir creation during setup

f94093c

fix isort

e6b5112

MarcoHuebner reviewed Nov 1, 2023

View reviewed changes

MarcoHuebner added 2 commits November 1, 2023 12:33

Improve clear_cache output for full wipe, remove unused import

724eb54

Address all non global-related pylint issues #20

a580d57

MarcoHuebner mentioned this pull request Nov 1, 2023

Feature/test coverage improval #4

Closed

pmayd and others added 3 commits November 4, 2023 00:34

because of complexity get rid of the current support of custom config…

8cd0d24

… dir and always use the default config dir under user home.

fix all tests; get rid of settings.ini and functionality for user to …

c754159

…define own config path; pystatis supports only default config path but custom data cache path

fix all tests; get rid of settings.ini and functionality for user to …

b2679da

…define own config path; pystatis supports only default config path but custom data cache path

pmayd added 4 commits November 4, 2023 22:04

refactor config module to work with a ConfigParser global config obje…

2859462

…ct instead of overwriting the config variable within the functions using global (bad style according to pylint)

address pylint issues

4a03be3

fix mypy issues

e88b8b6

fix pylint issues

14de704

pmayd merged commit c78967c into dev Nov 4, 2023
7 checks passed

pmayd deleted the feat/8-handle-multiple-databases-and-users branch November 4, 2023 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/8 handle multiple databases and users #20

Feat/8 handle multiple databases and users #20

pmayd commented Oct 27, 2023 •

edited

Loading

pmayd commented Oct 29, 2023

pmayd commented Oct 29, 2023

MarcoHuebner left a comment

MarcoHuebner Nov 1, 2023

pmayd Nov 1, 2023

MarcoHuebner Nov 4, 2023

MarcoHuebner Nov 1, 2023

pmayd Nov 1, 2023

MarcoHuebner Nov 1, 2023

pmayd Nov 1, 2023

MarcoHuebner Nov 1, 2023

MarcoHuebner Nov 1, 2023

pmayd commented Nov 4, 2023

MarcoHuebner commented Nov 4, 2023 •

edited

Loading

		"import logging\n",
		"logging.basicConfig(level=logging.INFO)\n",

Feat/8 handle multiple databases and users #20

Feat/8 handle multiple databases and users #20

Conversation

pmayd commented Oct 27, 2023 • edited Loading

pmayd commented Oct 29, 2023

pmayd commented Oct 29, 2023

MarcoHuebner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmayd commented Nov 4, 2023

MarcoHuebner commented Nov 4, 2023 • edited Loading

pmayd commented Oct 27, 2023 •

edited

Loading

MarcoHuebner commented Nov 4, 2023 •

edited

Loading