-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Feat/8 handle multiple databases and users (#20)
* change config module to handle multiple databases * finalize work on config module to handle multiple databases; significantly reduced lines of code by getting rid of the settings.ini * add a new db module that serves as a layer between the user and the config. Can set the current active database and get the settings from the config * simplify config module * refactor code to implement new config; correct tests * fix all remaining tests * fix all text issues * update notebooks according to latest changes in config * drop support for Python 3.9 due to pipe operator for types and set supported versions to 3.10 and 3.11 * fix problem with config dir creation during setup * fix isort * Improve clear_cache output for full wipe, remove unused import * Address all non global-related pylint issues #20 * because of complexity get rid of the current support of custom config dir and always use the default config dir under user home. * fix all tests; get rid of settings.ini and functionality for user to define own config path; pystatis supports only default config path but custom data cache path * fix all tests; get rid of settings.ini and functionality for user to define own config path; pystatis supports only default config path but custom data cache path * refactor config module to work with a ConfigParser global config object instead of overwriting the config variable within the functions using global (bad style according to pylint) * address pylint issues * fix mypy issues * fix pylint issues --------- Co-authored-by: MarcoHuebner <marco_huebner1@gmx.de>
- Loading branch information
1 parent
5813fb0
commit c78967c
Showing
26 changed files
with
846 additions
and
1,099 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Setup `pystatis`\n", | ||
| "\n", | ||
| "You don't need to do much to use `pystatis`. Basically, the first time you import the package, it will create a `config.ini` file under `~/.pystatis`. This file is used for storing settings, for example your credentials fpr the supported databases.\n", | ||
| "\n", | ||
| "To set up your credentials, we need to ask you for your username and password. This is done by the `setup_credentials()` function. It will ask you interactivly for the credentials, or you can set the following environmental variables:\n", | ||
| "- `PYSTATIS_GENESIS_API_USERNAME`\n", | ||
| "- `PYSTATIS_GENESIS_API_PASSWORD`\n", | ||
| "- `PYSTATIS_ZENSUS_API_USERNAME`\n", | ||
| "- `PYSTATIS_ZENSUS_API_PASSWORD`\n", | ||
| "- `PYSTATIS_REGIO_API_USERNAME`\n", | ||
| "- `PYSTATIS_REGIO_API_PASSWORD`" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "`dotenv` is uses here to load a local `.env` file that contains the above mentioned environmental variables so we don't have to input them." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "888706f5-3a9e-4e0a-9ca6-fa430280bc03", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from pathlib import Path\n", | ||
| "\n", | ||
| "import dotenv\n", | ||
| "\n", | ||
| "import pystatis\n", | ||
| "\n", | ||
| "print(\"pystatis version: \", pystatis.__version__)\n", | ||
| "dotenv.load_dotenv()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "ee7969b6", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# only execute if you want to delete your config file for test purposes\n", | ||
| "# config.delete_config()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "3928f347", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "`init_config` is called when loading pystatis, so a config with empty credentials will be created in your user home directory by default." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "18c2633f", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The only thing you have to do is to set up your user credentials.\n", | ||
| "\n", | ||
| "You can do so either by:\n", | ||
| "1. specifying the 4 environment variables `PYSTATIS_GENESIS_API_USERNAME|PASSWORD`, and `PYSTATIS_ZENSUS_API_USERNAME|PASSWORD`\n", | ||
| "2. calling the function `setup_credentials()` which will guide you through the process\n", | ||
| "\n", | ||
| "Even if you do 1. please call `setup_credentials()` once as it will read out the environment variables and write the credentials to the `config.ini` in your config dir." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "6c301da5", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "pystatis.setup_credentials() # also part of config module" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "d7d92f0d", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Once you have set up your credentials, they are stored in the `config.ini` and in the `config` object of the `config.py` module. You don't have to know this as regular user, this is more internal knowledge." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "**Warning**: The following code will print out the content of your `config.ini` file **with** the credentials set, so please do not share or push this notebook with outputs enabled." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "7ce103d9", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "with open(Path.home() / \".pystatis\" / \"config.ini\") as f:\n", | ||
| " print(f.read())" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.6" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Using different databases\n", | ||
| "\n", | ||
| "Once you have at least once called `setup_credentials()` you are good to use any method of this package with any of the supported databases.\n", | ||
| "\n", | ||
| "However, before you can do so, you have to tell `pystatis` which database you want to work with. \n", | ||
| "\n", | ||
| "This can be done with the `set_db()` function that is used to lock in the database for all subsequent called functions.\n", | ||
| "\n", | ||
| "If you don't know the names of the supported databases, ask `pystatis.config.get_supported_db()`." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from pystatis import set_db, logincheck, db\n", | ||
| "from pystatis import config" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "config.get_supported_db()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Whenever `pystatis` needs to fetch data from a database, it calls `get_db()` internally to get the database set is currently set as active database. If `set_db()` was not called before, `get_db()` will throw an error and inform you about this." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# expected to fail so you know you have to call set_db() first\n", | ||
| "db.get_db()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# expected to fail!\n", | ||
| "set_db(\"test\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "set_db(\"genesis\")\n", | ||
| "logincheck()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "set_db(\"zensus\")\n", | ||
| "logincheck()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "set_db(\"regio\")\n", | ||
| "logincheck()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "pystatis", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.6" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.