Configuration changes to make tests and import job run faster

❗	Warning: This page is going to be retired and this documentation will be maintained at - https://github.com/OpenConceptLab/ocl-docs/

Overview

OCLAPI's Common setting class dictates that every MongoDB update triggers an update in Solr's index. As the data grows this causes problems in running unit and integration tests as well as importing thousands of concepts.

Common settings do this via the setting below (excerpt from common.py)

HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

One simple workaround is to dictate that OCLAPI does not cause Solr to update its index and update the index when it is actually needed.

This is achieved by overriding HAYSTACK_SIGNAL_PROCESSOR setting in Dev class as in local.py

HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.BaseSignalProcessor'

Common scenarios:

• Run unit tests without updating the Solr index, Solr index is not needed during unit tests. Workaround cuts down time by 10x
• Run integration tests without updating the Solr index, update the index programmatically in a handful of tests where it's needed. Workaround cuts time by 10x
• Run an import job without updating the Solr index, run ./manage.py update_index afterwards. This cuts down the run time depending on the import size. Importing a few thousand concepts this way is done in about an hour instead of taking most of the day.

For importing concepts, an example would be like

docker-compose run -d --name ocl_import ocl_api bash -c "python manage.py import_concepts_to_source --source 58074380b80c4400145f2082 --token 355b37c48055ab284945f684f1c928d10da7b67e pih_concepts.json --configuration=Dev && echo y | python manage.py update_index