Add publisher status report notebook

[duncandewhurst]

@neelima-j I've documented the outstanding tasks for this PR below. Feel free to move these to issues if you want to get the PR merged. You don't need to do all of these before sharing the report with CoST, but it would be good to complete them before the project is wrapped up.

To do:

General

• Add new notebooks to the table in README.md
• Format SQL code in new notebooks using pgFormatter (can reuse the script from https://github.com/open-contracting/notebooks-ocds/blob/624dcca5d1c360132f0f50af7be03daa119b321a/manage.py#L165-L176)
• Ask tech infrastructure group about setting up database backups

Data import notebook

• Refactor so that all imports can be run at once

Quality criteria, checks and metrics notebook

• semantics_coordinates: Update to cover all supported geometry types
• criteria_registered: Add code to update registered_prefixes table from https://standard.open-contracting.org/staging/infrastructure/0.9-dev/en/reference/prefixes

Publisher status report notebook

• Replace collection_id in coverage query outputs with source_id. See get_output function definition for an example.
• Format coverage query outputs for readability using Styler, e.g. colour scale for coverage scores, collapsing/expanding for objects. See get_results for an example of how to do this.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[duncandewhurst]

@neelima-j please see the list of outstanding tasks in the PR description.

[neelima-j]

Outstanding tasks moved to separate issues #45 #46 #47

[odscjen]

For each data source, choose a data source and run after (Ctrl+F10)

(Ctrl+F10) doesn't run the cell for me. I suspect this might be a browser specific thing (I use Firefox), or maybe it's to do with the other extensions I've got running in my browser, but either way as this isn't a universal command I'd remove it and replace it with a command to run the cell (which I assume it what it's supposed to do?)

Reply via ReviewNB

[odscjen]

Using run_id = 2024-08-02 06:29:25.083245

Cell didn't run, got the error

File "<ipython-input-20-704129bd1c51>", line 5
    source_id,
    ^
IndentationError: unexpected indent

Reply via ReviewNB

[odscjen]

Line #3.    % % sql

should be %%sql i.e. need to remove the spaces

Reply via ReviewNB

[odscjen]

cell returned error

KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py in get_loc(self, key)
   3804         try:
-> 3805             return self._engine.get_loc(casted_key)
   3806         except KeyError as err:

index.pyx in pandas._libs.index.IndexEngine.get_loc()

index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'check'
The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)

6 frames

/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py in get_loc(self, key)
   3810             ):
   3811                 raise InvalidIndexError(key)
-> 3812             raise KeyError(key) from err
   3813         except TypeError:
   3814             # If we have a listlike key, _check_indexing_error will raise

KeyError: 'check'

Reply via ReviewNB

[odscjen]

This and the following other date cells, in the returned table the column with the date is titled "count" which is confusing

Reply via ReviewNB

[odscjen]

I don't understand why 2 collections need to be chosen and what the difference should be between them, some explanation is needed, or if there is explanation elsewhere it needs to be signposted here

Reply via ReviewNB

[odscjen]

This is returning an empty table which means all the other checks also return an empty table, not sure what I'm doing wrong? I used 'dev_1' and 'dev_2' as the default load and comparison_load ids and then also tried with 2 collections I created myself using the import notebook with the same empty result.

Reply via ReviewNB