Adds CSV loader for SearchEvents #54
Conversation
JPrevost
force-pushed
the
tco35-csv-data-loader
branch
from
1cf9848
to
bfd0b7c
Compare
|
|
||
| CSV.foreach(args.path) do |row| | ||
| term = Term.create_or_find_by!(phrase: row.first) | ||
| term.search_events.create!(source: args.source, created_at: row.last) |
There was a problem hiding this comment.
It doesn't matter that this will create duplicate search events if, for example, the same CSV is loaded twice, right? I assume not given that this is just for dev environments, but I want to make sure I'm not missing anything.
There was a problem hiding this comment.
@JPrevost Tagging on ☝️, as it occurred to me that GitHub may not send notifications on comments until a review is complete.
There was a problem hiding this comment.
You are correct that this would load the same csv multiple times if run multiple times and doesn't try to detect that it has already been loaded. I suspect this is fine, but maybe at minimum we should note?
There was a problem hiding this comment.
I just pushed a note to that effect. Will approve now so you can merge this as soon as you like, but feel free to modify the note as needed. (I'm not sure what multiple notes for the same method look like in YARD, and for some reason my local docserver isn't showing me the CSV loader task.)
There was a problem hiding this comment.
Yeah, the tasks don't show up in Yard sadly. I suspect we can nudge them in there with some config. Multiple notes are fine though from my poking around on other methods.
|
|
||
| CSV.foreach(args.path) do |row| | ||
| term = Term.create_or_find_by!(phrase: row.first) | ||
| term.search_events.create!(source: args.source, created_at: row.last) |
There was a problem hiding this comment.
I just pushed a note to that effect. Will approve now so you can merge this as soon as you like, but feel free to modify the note as needed. (I'm not sure what multiple notes for the same method look like in YARD, and for some reason my local docserver isn't showing me the CSV loader task.)
https://mitlibraries.atlassian.net/browse/TCO-35 This loader is not meant to be used in production, but instead be a convenience for loading data into development environments to allow for easier development of detection algorithms or reporting. There is a Dataclip that exports data in the expected format, but it is also easy to generate your own file as it is just `term` and `timestamp` that is expected. Currently, `source` is provided via CLI but if we find it would be better to include the source in the CSV that would be an easy change to make.
JPrevost
force-pushed
the
tco35-csv-data-loader
branch
from
ab8298c
to
a24e2ef
Compare
https://mitlibraries.atlassian.net/browse/TCO-35
This loader is not meant to be used in production, but instead be a convenience for loading data into development environments to allow for easier development of detection algorithms or reporting.
There is a Dataclip that exports data in the expected format, but it is also easy to generate your own file as it is just
termandtimestampthat is expected.Currently,
sourceis provided via CLI but if we find it would be better to include the source in the CSV that would be an easy change to make.