Skip to content

Commit

Permalink
Documentation of term-search-basic lambda
Browse files Browse the repository at this point in the history
  • Loading branch information
mikejcorey committed Jul 25, 2024
1 parent 2b766f1 commit b9eaaad
Show file tree
Hide file tree
Showing 3 changed files with 52 additions and 1 deletion.
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ Deed Machine full workflow

modules/lambdas/mp-covenants-split-pages.rst
modules/lambdas/mp-covenants-ocr-page.rst
modules/lambdas/mp-covenants-term-search-basic.rst

.. toctree::
:maxdepth: 2
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/components.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The individual lambda functions that make up the OCR, term search and web image

- :ref:`mp-covenants-split-pages`
- :ref:`mp-covenants-ocr-page`
- :ref:`mp-covenants-term-search-basic`

- `mp-covenants-term-search-basic <https://github.com/UMNLibraries/mp-covenants-term-search-basic>`_
- `mp-covenants-resize-image <https://github.com/UMNLibraries/mp-covenants-resize-image>`_
- `mp-covenants-fake-ocr <https://github.com/UMNLibraries/mp-covenants-fake-ocr>`_
50 changes: 50 additions & 0 deletions docs/modules/lambdas/mp-covenants-term-search-basic.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
.. _mp-covenants-term-search-basic:

mp-covenants-term-search-basic
===============================

- Code: `mp-covenants-term-search-basic on Github <https://github.com/UMNLibraries/mp-covenants-term-search-basic>`_

This component looks for racial and other terms to flag as potential racial covenants. For each term in covenant_flags, check OCR JSON file received from previous step for existance of term. This is a really simple CNTRL + F style search, no partial matching, regex, fuzzy, etc. Some of the terms are actually exceptions rather than covenant hits, and once they reach the Django stage, will be used to mark this page as exempt from consideration as being considered as a racial covenant. Common examples of exceptions include birth certificates and military discharges, which often contain racial information but are not going to contain a racial covenant.. This is the third Lambda in the Deed Machine initial processing Step Function.


Steps of the function
---------------------

1. Check event for valid image.
2. For each term, search each line of text for term
3. If term found, add to output JSON along with which line numbers it appears in.
4. Return path to match file.

Software development requirements
---------------------------------

The Lambda components of the Deed Machine are built using Amazon's Serverless Application Model (SAM) and the AWS SAM CLI tool.

- Pipenv (Can use other virtual environments, but will require fiddling on your part)
- AWS SAM CLI
- Docker
- Python 3

Quickstart commands
-------------------

To build the application:

.. code-block:: bash
pipenv install
pipenv shell
sam build
To rebuild and deploy the application:

.. code-block:: bash
sam build && sam deploy
To run tests:

.. code-block:: bash
pytest

0 comments on commit b9eaaad

Please sign in to comment.