Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add publisher status report notebook #43

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Add publisher status report notebook #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
1c26c7c
Add publisher status report notebook
duncandewhurst Jul 29, 2024
4ad33fb
Publisher Status Report: Add link to data import notebook
duncandewhurst Jul 29, 2024
88304b6
Add quality criteria, checks and metrics notebook
duncandewhurst Jul 29, 2024
304c24d
Merge pull request #44 from open-contracting/main
duncandewhurst Jul 29, 2024
57b49a6
data import notebook: add load_id, refactor
duncandewhurst Jul 29, 2024
0c4b846
Update data import notebook
duncandewhurst Jul 30, 2024
1a08076
quality criteria, checks and metrics: Add coverage checks
duncandewhurst Aug 1, 2024
1a4cbb2
Publisher status report: Add coverage section
duncandewhurst Aug 1, 2024
5ad93de
Publisher status report: Add coverage docs
duncandewhurst Aug 1, 2024
7f3da3f
README.md: Add links to quality check and status report notebooks
neelima-j Sep 16, 2024
4913575
Replace collection_id in coverage query outputs with source_id, copy …
neelima-j Sep 17, 2024
1db6e17
Publisher status report: Format coverage query outputs for readability
neelima-j Sep 17, 2024
e44f007
Quality criteria checks and metrics: Update semantics_coordinates to …
neelima-j Sep 18, 2024
6a75701
Fix spelling of 'criteria' in text and code
neelima-j Oct 7, 2024
2fcaf65
Formats SQL code using pgFormatter
neelima-j Oct 7, 2024
201494e
Formats SQL code using pgFormatter
neelima-j Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 99 additions & 76 deletions OC4IDS_Database_Data_Import.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@
"\n",
Copy link
Contributor

@odscjen odscjen Oct 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For each data source, choose a data source and run after (Ctrl+F10)

(Ctrl+F10) doesn't run the cell for me. I suspect this might be a browser specific thing (I use Firefox), or maybe it's to do with the other extensions I've got running in my browser, but either way as this isn't a universal command I'd remove it and replace it with a command to run the cell (which I assume it what it's supposed to do?)


Reply via ReviewNB

"**How to run the notebook**\n",
"\n",
"1. [Choose a data source](#scrollTo=HFTcVz0Q0tMr)\n",
"1. Run all cells in the notebook (`Ctrl+F9`)\n",
"1. [Enter your database credentials](#scrollTo=z4-iWuZRhoEe) at the prompt.\n",
"1. [Enter your database credentials](#scrollTo=z4-iWuZRhoEe)\n",
"1. Run all cells in the [Setup](#scrollTo=3PU3KAsPuYP7) section\n",
"1. For each load (set of data sources that you wish to analyze as a group):\n",
" 1. [Set a new `load_id`](#scrollTo=XJSBAcLbWJHu)\n",
" 1. For each data source, [choose a data source](#scrollTo=HFTcVz0Q0tMr) and run after (Ctrl+F10)\n",
"\n",
"**How to add a new publisher**\n",
"\n",
Expand All @@ -32,27 +34,6 @@
"[Open an issue in the `notebooks-oc4ids` repository](https://github.com/open-contracting/notebooks-oc4ids/issues/new)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HFTcVz0Q0tMr"
},
"source": [
"## Choose a data source"
]
},
{
"cell_type": "code",
"metadata": {
"id": "zMP_MPYzycvD",
"cellView": "form"
},
"source": [
"source_id = 'uganda_gpp' #@param [ 'mexico_cost_jalisco', 'ghana_cost_sekondi_takoradi', 'mexico_nuevo_leon', 'indonesia_cost_west_lombok', 'ukraine_cost_ukraine', 'uganda_gpp', 'malawi_cost_malawi']"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -81,56 +62,6 @@
"password = getpass.getpass('Password:')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "veSGp6SIwYRt"
},
"source": [
"## Download data"
]
},
{
"cell_type": "code",
"metadata": {
"id": "SHbJftL1zkMK"
},
"source": [
"import json\n",
"import requests\n",
"\n",
"from datetime import datetime\n",
"\n",
"sources = {\n",
" 'mexico_cost_jalisco': 'http://www.costjalisco.org.mx/jsonprojects',\n",
" 'ghana_cost_sekondi_takoradi': 'https://costsekondi-takoradigh.org/uploads/projectJson.json',\n",
" 'mexico_nuevo_leon': 'http://si.nl.gob.mx/siasi_ws/api/edcapi/DescargarProjectPackage',\n",
" 'indonesia_cost_west_lombok': 'https://intras.lombokbaratkab.go.id/oc4ids',\n",
" 'ukraine_cost_ukraine': 'https://portal.costukraine.org/data.json',\n",
" 'uganda_gpp': 'https://gpp.ppda.go.ug/adminapi/public/api/open-data/v1/infrastructure/projects/download?format=json',\n",
" 'malawi_cost_malawi': 'https://ippi.mw/api/projects/query'\n",
"}\n",
"\n",
"if source_id == 'malawi_cost_malawi':\n",
" # The IPPI API accepts a POST request with JSON-encoded start_date and end_date paramters\n",
" payload = {\n",
" \"start_date\": \"2010-01-01\",\n",
" \"end_date\": datetime.today().strftime('%Y-%m-%d')\n",
" }\n",
" response = requests.post(sources[source_id], json=payload)\n",
"\n",
" with open('project_package.json', 'wb') as f:\n",
" f.write(response.content)\n",
"\n",
"else:\n",
" response = requests.get(sources[source_id], verify=False)\n",
"\n",
" with open('project_package.json', 'wb') as f:\n",
" f.write(response.content)\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -267,6 +198,98 @@
"pip install libcoveoc4ids"
]
},
{
"cell_type": "markdown",
"source": [
"## Set a new `load_id`"
],
"metadata": {
"id": "XJSBAcLbWJHu"
}
},
{
"cell_type": "code",
"source": [
"from datetime import datetime\n",
"load_id = datetime.now()"
],
"metadata": {
"id": "AIGD4qmoWMOD"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "HFTcVz0Q0tMr"
},
"source": [
"## Choose a data source"
]
},
{
"cell_type": "code",
"metadata": {
"id": "zMP_MPYzycvD",
"cellView": "form"
},
"source": [
"source_id = 'uganda_gpp' #@param [ 'mexico_cost_jalisco', 'ghana_cost_sekondi_takoradi', 'mexico_nuevo_leon', 'indonesia_cost_west_lombok', 'ukraine_cost_ukraine', 'uganda_gpp', 'malawi_cost_malawi']"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "veSGp6SIwYRt"
},
"source": [
"## Download data"
]
},
{
"cell_type": "code",
"metadata": {
"id": "SHbJftL1zkMK"
},
"source": [
"import json\n",
"import requests\n",
"\n",
"from datetime import datetime\n",
"\n",
"sources = {\n",
" 'mexico_cost_jalisco': 'http://www.costjalisco.org.mx/jsonprojects',\n",
" 'ghana_cost_sekondi_takoradi': 'https://costsekondi-takoradigh.org/uploads/projectJson.json',\n",
" 'mexico_nuevo_leon': 'http://si.nl.gob.mx/siasi_ws/api/edcapi/DescargarProjectPackage',\n",
" 'indonesia_cost_west_lombok': 'https://intras.lombokbaratkab.go.id/oc4ids',\n",
" 'ukraine_cost_ukraine': 'https://portal.costukraine.org/data.json',\n",
" 'uganda_gpp': 'https://gpp.ppda.go.ug/adminapi/public/api/open-data/v1/infrastructure/projects/download?format=json',\n",
" 'malawi_cost_malawi': 'https://ippi.mw/api/projects/query'\n",
"}\n",
"\n",
"if source_id == 'malawi_cost_malawi':\n",
" # The IPPI API accepts a POST request with JSON-encoded start_date and end_date paramters\n",
" payload = {\n",
" \"start_date\": \"2010-01-01\",\n",
" \"end_date\": datetime.today().strftime('%Y-%m-%d')\n",
" }\n",
" response = requests.post(sources[source_id], json=payload)\n",
"\n",
" with open('project_package.json', 'wb') as f:\n",
" f.write(response.content)\n",
"\n",
"else:\n",
" response = requests.get(sources[source_id], verify=False)\n",
"\n",
" with open('project_package.json', 'wb') as f:\n",
" f.write(response.content)\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -435,8 +458,8 @@
"source": [
"%%sql\n",
"\n",
"INSERT INTO collection (source_id, data_version)\n",
" VALUES (:source_id, CURRENT_TIMESTAMP);\n",
"INSERT INTO collection (source_id, data_version, load_id)\n",
" VALUES (:source_id, CURRENT_TIMESTAMP, :load_id);\n",
"\n",
"INSERT INTO projects (collection_id, project_id, data)\n",
"SELECT\n",
Expand Down
Loading