Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make OGM tasks more production-ready #169

Merged
merged 4 commits into from
Feb 28, 2024
Merged

Conversation

thatbudakguy
Copy link
Member

@thatbudakguy thatbudakguy commented Feb 23, 2024

This admittedly large PR happened because I wanted to set up regularly scheduled updates from OpenGeoMetadata for Earthworks via cron, and GeoCombine seemed like the best way to do it (sul-dlss/earthworks#639).

It tackles several outstanding issues:

@thatbudakguy
Copy link
Member Author

Here's an example of what you see when indexing. I modified a document to have some bad metadata as an example.

W, [2024-02-23T14:18:28.933869 #41378]  WARN -- GeoCombine: SOLR_URL not set; using Blacklight default
I, [2024-02-23T14:18:28.936742 #41378]  INFO -- GeoCombine: indexing into http://localhost:8983/solr/blacklight-core
I, [2024-02-23T14:18:28.936772 #41378]  INFO -- GeoCombine: loading documents from tmp/opengeometadata
E, [2024-02-23T14:19:01.621359 #41378] ERROR -- GeoCombine: error indexing batch (100 docs): 400 Bad Request - ERROR: [doc=VAC3073-M-01154] Error adding field 'solr_geom'='fdjlskfjdlkjfw' msg=Unable to parse shape given formats "lat,lon", "x y" or as WKT because java.text.ParseException: Unknown Shape definition [fdjlskfjdlkjfw]
W, [2024-02-23T14:19:01.621400 #41378]  WARN -- GeoCombine: retrying documents individually
E, [2024-02-23T14:19:01.675005 #41378] ERROR -- GeoCombine: error indexing tmp/opengeometadata/fake/geoblacklight.json: 400 Bad Request - ERROR: [doc=VAC3073-M-01154] Error adding field 'solr_geom'='fdjlskfjdlkjfw' msg=Unable to parse shape given formats "lat,lon", "x y" or as WKT because java.text.ParseException: Unknown Shape definition [fdjlskfjdlkjfw]
I, [2024-02-23T14:19:03.313709 #41378]  INFO -- GeoCombine: indexed 10084 documents in 34.38 seconds

Copy link
Collaborator

@dl-maura dl-maura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked on my local, confirmed behavior in slack

@thatbudakguy thatbudakguy merged commit 672076a into main Feb 28, 2024
7 checks passed
@thatbudakguy thatbudakguy deleted the production-indexing branch February 28, 2024 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants