Skip errors during harvest and report to Sentry

[ghukill]

What does this PR do?

Allows graceful skip of errors during method=get / GetRecords style harvest.

If errors are encountered, and the count is below a new config value MAX_ALLOWED_ERRORS, a message is sent to Sentry that looks roughly like:

However, if more errors are encountered than defined by MAX_ALLOWED_ERRORS, the harvested is stopped and a message like the following will be sent to Sentry:

In both scenarios, an additional data point failed_records will be present in the Sentry message that shows which records failed, e.g.:

Helpful background context

As noted in both TIMX-257 and TIMX-258 (linked below), until now an external "skip list" has been used to skip records that were consistently throwing errors. This was functional, but has some overhead for maintaining that list; more detail can be found in the tickets.

How can a reviewer manually see the effects of these changes?

The most straightforward way is to perform a local harvest that targets some known (at time of writing) problematic records:

from harvester.oai import OAIClient

oai_client = OAIClient(
    "https://dspace.mit.edu/oai/request",
    metadata_format="oai_dc",
    retry_status_codes=tuple() # sets to empty tuple to avoid numerous retries
)
identifiers = [
    "oai:dspace.mit.edu:1721.1/152958",
    "oai:dspace.mit.edu:1721.1/152786", # BAD
    "oai:dspace.mit.edu:1721.1/147573", # BAD
    "oai:dspace.mit.edu:1721.1/152939",
]
records = list(oai_client.get_records(identifiers))

This should complete, but records should only be length 2 for the two that completed successfully.

To confirm that messages are sent to Sentry, you can do the following:

• set .env

WORKSPACE=dev
SENTRY_DSN=<ADD HERE>

And then prefix the following to python code above:

from harvester.config import configure_sentry
configure_sentry()

# rest of code...

After re-running, there should be a new message here in Sentry.

Includes new or updated dependencies?

YES

NOTE: both vcrpy and urllib3 was pinned until work is completed for this issue: #550

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-257
• https://mitlibraries.atlassian.net/browse/TIMX-258

Developer checklist

• All new ENV is documented in README
• Stakeholder approval has been confirmed (or is not needed)

Code reviewer checklist

• The commit message is clear and follows our guidelines (not just this pull request message)
• There are appropriate tests covering any new functionality
• The documentation has been updated or is unnecessary
• The changes have been verified
• New dependencies are appropriate or there were no changes

[ehanson8]

Looks good and works as expected, one suggestion!

[github-actions]

Pull Request Test Coverage Report for Build 6931383261

• 32 of 32 (100.0%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 4670459934:	0.0%
Covered Lines:	179
Relevant Lines:	179

---

💛 - Coveralls