OpenMRS Concept Dictionary Import

Current Process

1. Download zipped OpenMRS v1.11.x concept dictionary from Dropbox (e.g. openmrs_concepts_1.11.4_20180601.sql.zip)
2. Unzip and import into MySQL
   • In terminal: unzip openmrs_concepts_1.11.4_20180601.sql.zip
   • In mysql shell: create database ciel_20180601
   • In terminal: mysql -u username -p ciel_20180601 < openmrs_concepts_1.11.4_20180601.sql
   • Note that you can ignore the "global property" if it displays: ERROR 1146 (42S02) at line 1872: Table 'ciel_20180601.global_property' doesn't exist
3. Verify that required map sources exist on the target OCL server:
   • In the ocl_omrs repo, configure settings.py with the MySQL database, username, and password for the database containing the OpenMRS concept dictionary (e.g. ciel_20180601)
   • In the ocl_omrs repo, run this command: python manage.py extract_db --check_sources --env=... --token=... where is qa, staging, or production and token is the API token of a user that has at least read-only access to the necessary organizations and sources
   • If any organizations or sources are missing, then these currently need to be generated manually and also manually add the new orgs/sources to __init__.py
4. Generate the OCL-formatted import files, one for concepts and one for mappings
   • In the ocl_omrs repo, run the commands to extract the concepts andd mappings in raw OCL-formatted JSON:

python manage.py extract_db --org_id=CIEL --source_id=CIEL --raw -v0 --concepts > concepts.json
python manage.py extract_db --org_id=CIEL --source_id=CIEL --raw -v0 --mappings > mappings.json

5. On the OCL server, import the concepts and mappings (to staging and production)
   • Documentation was pasted below and is maintained here: https://github.com/OpenConceptLab/oclapi#importing-ciel-to-staging-and-production

In order to import a newer version of the CIEL dictionary you need to have an SSH root access to staging.openconceptlab.org and openconceptlab.org. Download the zip file with concepts and mappings in the OCL format and run the following commands for staging:

sudo -s
cd /root/docker/oclapi-stg 
unzip /path/to/zip/ciel_20180223.zip
docker-compose run -d --rm -v /root/docker/oclapi-stg:/ciel api python manage.py import_concepts_to_source --source 57cd60e2ba0d489c55039465 --token REPLACE_WITH_ROOT_API_TOKEN --retire-missing-records /ciel/ciel_20180223_concepts.json
docker logs -f oclapistg_api_run_1
docker-compose run -d --rm -v /root/docker/oclapi-stg:/ciel api python manage.py import_mappings_to_source --source 57cd60e2ba0d489c55039465 --token REPLACE_WITH_ROOT_API_TOKEN --retire-missing-records /ciel/ciel_20180223_mappings.json
docker logs -f oclapistg_api_run_2

Or for production:

sudo -s
cd /root/docker/oclapi-prd 
unzip /path/to/zip/ciel_20180223.zip
docker-compose run -d --rm -v /root/docker/oclapi-prd:/ciel api python manage.py import_concepts_to_source --source 5821b7a564d700001440f44a --token REPLACE_WITH_ROOT_API_TOKEN --retire-missing-records /ciel/ciel_20180223_concepts.json
docker logs -f oclapiprd_api_run_1
docker-compose run -d --rm -v /root/docker/oclapi-prd:/ciel api python manage.py import_mappings_to_source --source 5821b7a564d700001440f44a --token REPLACE_WITH_ROOT_API_TOKEN --retire-missing-records /ciel/ciel_20180223_mappings.json
docker logs -f oclapiprd_api_run_2

Imports run in background so you can disconnect from the server any time, but note that you must wait for concepts to be imported before importing mappings. You can get back to logs at any point by running: docker logs -f CONTAINER_NAME.

Desired Process

API Request:

• Post zipped OpenMRS v1.11.x concept dictionary to OCL API endpoint (e.g. https://api.openconceptlab.org/omrs_import/)
  • The OCL API will accept a zipped SQL file representing a MySQL dump of an OpenMRS v1.11.x concept dictionary
  • Note that in the future, we may want the omrs_import endpoint to support OpenMRS dictionary versions other than v1.11.x
  • The API request will perform all the manual steps that currently take place:
    • Accept a zipped SQL file representing a MySQL dump of an OpenMRS v1.11.x concept dictionary --- or alternatively consider having the user post a URL and have the server do a wget, instead of uploading a quite large zip file
    • Verify that the org/source exist and that the user has appropriate access to perform the import
    • Unzip the SQL file and import it into a local temporary database
    • Validate that the OMRS dictionary is syntactically correct?
    • Verify that the map sources used in the OpenMRS dictionary are defined in OCL -- see issue below -- and handle any missing orgs/sources. This probably means failing with an error status code and a response body that contains details of the missing orgs and sources -- unless it can be handled automatically?
    • Generate import files
    • Document the
    • Import the files
    • ?? Run a diff to show what will be changed?
    • Generate a new source version and (optionally) release
  • The endpoint needs the following parameters:
    • Zipped sql concept dictionary
    • API Token for the user account to perform the import
    • Owner/Source to import into
    • ID of Source version to create
    • Whether to mark the new source version as released

Response

TBD

Issues to be resolved:

• Importing the OMRS concept dictionary into MySQL and verifying that it is valid and that the map sources all exist can take some time -- usually a few minutes on my local computer -- this may mean that we need to run imports asynchronously with additional API endpoints to check on the status of an import
• Map source identifiers in OpenMRS are not all the same in OCL -- the mappings between the OCL and OpenMRS/CIEL IDs are currently hard coded, but a more generic method is needed