parleh

Python scripts to download parliamentary data from offical sources.

Supported sources

This project uses pipenv to manage the Python application environment and required packages (e.g. requests, pandas). For details, see https://packaging.python.org/en/latest/tutorials/managing-dependencies/. Steps:

On Mac, ensure pipenv is installed using brew install pipenv.
To switch the command line (shell) to the project's Python environment: pipenv shell.
To install packages (already done): pipenv install pandas requests.

visit https://lop.parl.ca/sites/ParlInfo/default/en_CA/People/parliamentarians
take note of the latest parliament number and whether "Currently in Office" is shown
in ca directory:
- edit config.py and update the start_parl, end_parl numbers and the include_current flag accordingly
- in shell, run make clean all which does:
  - downloads the people (parliamentarians) in each parliament to data/parliaments/parliament-N-people.json
  - generates a corresponding, simplified CSV file for each: data/parliaments/parliament-N-people.csv
  - generates a combined, simplified CSV file: data/parliaments/all_parliaments.csv
  - downloads the full person record for each parliamentarian to data/people/ID-LAST_NAME,FIRST_NAME.json
  - compresses the above into two zip files: data/parliaments/parliaments.zip and data/people/people.zip

list of parliaments: curl -H "Accept: application/json" "https://lop.parl.ca/ParlinfowebAPI/Parliament/GetParliamentSessionSittingList" | jq .
"refiners" (search options) for parliamentarians: curl -H "Accept: application/json" "https://lop.parl.ca/ParlinfoWebAPI/Refiner/GetRefiners?collection=Person" | jq .
- note that the refiner with "RefinerId": 4 selects which parliament(s) to include, but the listed options are in reverse order
- e.g. "OptionId": 1 is for "Currently in Office", while "OptionId": 2 is for the most recent parliament, and "OptionId": 3 is for the previous parliament