Samples data #741

ldecicco-USGS · 2024-12-21T15:43:02Z

No description provided.

seeing how readme gets rendered without image (we'll try to get it back though!)

… on GL pages.

…ile.

… samples_data

…gered manually. I think there are some rate-limiting features going on in the CI job

Fix WQP siteInfo attribute bug

Merge branch 'main' of https://github.com/DOI-USGS/dataRetrieval into samples_data # Conflicts: # R/readWQPdata.R

ehinman

Cool MR! Left some comments, but generally find the new functions and vignette easy to follow and use.

Weird observation, but I ran this:
test <- read_USGS_samples(pointLocationLatitude = 43.1206415, pointLocationLongitude = -89.3691616, pointLocationWithinMiles = 15, activityStartDateLower = "2024-10-01")

And it failed. Then I looked at the vignette and matched the vignette's example:
test <- read_USGS_samples(pointLocationLatitude = 43.1206415, pointLocationLongitude = -89.3691616, pointLocationWithinMiles = 15, activityStartDateLower = "2024-10-01", dataType = "Monitoring locations", dataProfile = "Site")

And it worked. Then I tried the first query again, and it worked. Not sure if the ordering of the queries matters at all, but I found it peculiar enough to share. Could have totally been coincidence.

ehinman · 2025-01-02T15:21:39Z

tests/testthat/tests_samples.R

+
+
+context("samples-data samples")
+test_that("samples-data samples working", {


Suggested change

test_that("samples-data samples working", {

test_that("samples-data activities working", {

ehinman · 2025-01-02T15:23:09Z

tests/testthat/tests_samples.R

+
+  expect_true(all(c(
+    "Org_Identifier",
+    "Org_FormalName"


Assuming you pick these columns because they are the first ones in the returned df if completed successfully/as expected?

Yeah. Could be more thorough, but that's it.

ehinman · 2025-01-02T15:25:09Z

tests/testthat/tests_nldi.R

+#              no_sf = TRUE,
+#              warn = TRUE)
+#   )
+# })


Why commented out?

NLDI recently switched to api.gov (something like that at least). That means there's now rate limited...which means if I hit the same request a bunch of times, it errors out. We're going to have to figure out how to deal with this eventually (both for NLDI and all future new services)....but that's a problem for another day. In the meantime, Dave recommended commenting this out for now

@ldecicco-USGS can we create an Issue to track this? Just to make sure future selves remember.

ehinman · 2025-01-02T15:27:45Z

R/readUSGSsample.R

+#'                                dataType = "Projects")
+#' 
+#' }
+read_USGS_samples <- function(monitoringLocationIdentifier = NA,


Minor, but suggest naming file read_USGS_samples.R.

yeah, decided on the snake case after starting the work. Renaming...

ehinman · 2025-01-02T15:28:54Z

R/readUSGSsample.R

+#' See also: \url{https://api.waterdata.usgs.gov/samples-data/docs}.
+#'  
+#' @param monitoringLocationIdentifier A monitoring location identifier has two parts: the agency code
+#' and the location number. Location identifiers should be separated with commas,


Suggested change

#' and the location number. Location identifiers should be separated with commas,

#' and the location number, separated by a dash (-). Location identifiers should be separated with commas,

ehinman · 2025-01-02T17:41:14Z

vignettes/samples_data.Rmd

+
+## Return data types
+
+There are 2 arguments that dictate what kind of data is returned: dataType and dataProfile. The "dataType" argument define what kind of data comes back, and the "dataProfile" define what columns from that type come back.


Suggested change

There are 2 arguments that dictate what kind of data is returned: dataType and dataProfile. The "dataType" argument define what kind of data comes back, and the "dataProfile" define what columns from that type come back.

There are 2 arguments that dictate what kind of data is returned: dataType and dataProfile. The "dataType" argument defines what kind of data comes back, and the "dataProfile" defines what columns from that type come back.

ehinman · 2025-01-02T17:55:29Z

R/readUSGSsample.R

+#' with pointLocationLatitude and pointLocationLongitude
+#' @param dataType Options include: "Results", "Monitoring locations", "Activities",
+#' "Projects", and "Organizations".
+#' @param dataProfile Profile depends on type. Options for "Results" dataType are: 


Missing some? From Samples API: Available values: fullphyschem, basicphyschem, fullbio, basicbio, narrow, resultdetectionquantitationlimit, labsampleprep, count

I guess you're matching what's here: https://waterdata.usgs.gov/download-samples/#dataProfile=fullphyschem ?

ehinman · 2025-01-02T18:23:16Z

R/readUSGSsample.R

+#' "Site" and "Count". Options for "Activities" are "Sample Activities", 
+#' "Activity Metrics", "Activity Groups", "Count". Options for "Projects" are:
+#' "Project" and "Project Monitoring Location Weight". Options for "Organizations" are:
+#' "Organization" and "Count".


I appreciate matching the services and profiles with the Samples download interface, but I also would prefer to simply match the actual service names as outlined here: https://api.waterdata.usgs.gov/samples-data/docs#/ It's sometimes onerous to remember what is capitalized, what is not, where there are spaces, etc. I admit some impatience here, but I'd prefer the inputs to match what actually shows up in the URL generated at the bottom of the Samples GUI. That also saves you a little bit of effort in the code as well.

ehinman · 2025-01-02T18:44:55Z

R/readUSGSsample.R

+#' @param characteristic Characteristic is a specific category describing the sample.
+#' See available options by running 
+#' \code{check_param("characteristics")$characteristicName}.
+#' @param stateFips State query parameter. To get a list of available state fips, 


You have a nice example of this in the vignette, but would be helpful to show the general format here as well: "FIPs codes for states take the format: CountryAbbrev:StateNumber, like US:55 for Wisconsin." I got a little turned around trying to write my own function query. First I went to check_params("state") and then wasn't sure why I'd also need to do stateCdLookup(), but then realized I needed the correct fips format, which can be extracted using stateCdLookup(), but I needed to look at the help to understand that the correct function parameterization is stateCdLookup("WI", "fips").

ehinman · 2025-01-02T18:48:24Z

R/readUSGSsample.R

+#' \code{stateCdLookup}.
+#' @param countyFips County query parameter. To get a list of available counties,
+#' run \code{check_param("counties")}. The "Fips" can be created using the function
+#' \code{countyCdLookup}.


You have a nice example of this in the vignette, but same general comment as above, in case someone does not go to the vignette.

p.s. in all my muddling through this, the errors were very helpful, and I eventually got to where I needed. So that is definitely a nice aspect of this whole system of helper functions and lookup tables!

… into samples_data

ldecicco-USGS added 30 commits November 8, 2024 10:04

convert both legacy and new columns

7ad8b0d

Merge branch 'main' of https://github.com/DOI-USGS/dataRetrieval

13ab814

First pass at samples-data

e2fe792

Adding some verification

536e9c1

More samples-data updates

afef6e5

Fix examples

8b1d407

point within

0667ec1

add more services

443d587

Update README.md

e27395d

seeing how readme gets rendered without image (we'll try to get it back though!)

Adding sample-data tests

866100c

Only run verification on main branch, but do create development pages…

17fd956

… on GL pages.

Rejigger state/county code lookups

4a75a03

Don't think we need to rebuild the image each time we change the ci f…

c920ec0

…ile.

Cleaning up a few more argument descriptions.

e72c7aa

Fix some tests

8d81e41

start a vignette

4683cb7

get pkgdown working

a7076d7

Always causing problems

1efac0d

Update vignette

003e8cd

Merge branch 'main' of https://github.com/DOI-USGS/dataRetrieval into…

e8188e4

… samples_data

another wade causing trouble

09cfb45

moving all nldi tests to "long running", so should only run when trig…

9957587

…gered manually. I think there are some rate-limiting features going on in the CI job

Adding more text.

46d2a32

Switch to summarize

8c58d4c

Updates from Lee's review

126f077

trying to avoid the longrunner

89d54de

fix siteInfo bug on list arguments

46860bc

renamed summarize

7b5d81f

Merge pull request DOI-USGS#739 from ldecicco-USGS/main

573da06

Fix WQP siteInfo attribute bug

upstream pull

e9c188e

Merge branch 'main' of https://github.com/DOI-USGS/dataRetrieval into samples_data # Conflicts: # R/readWQPdata.R

ldecicco-USGS added 2 commits December 4, 2024 09:53

Cleanup mapping code

d15ee15

Ignore samples_data in package

3cb309c

ldecicco-USGS requested a review from ehinman December 23, 2024 13:10

ehinman approved these changes Jan 2, 2025

View reviewed changes

ldecicco-USGS added 2 commits January 15, 2025 14:42

Response to Elise's review.

670924d

Merge branch 'samples_data' of github.com:ldecicco-USGS/dataRetrieval…

dc71fc4

… into samples_data

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Samples data #741

Samples data #741

ldecicco-USGS commented Dec 21, 2024

ehinman left a comment •

edited

Loading

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ldecicco-USGS Jan 15, 2025

ehinman Jan 2, 2025

ldecicco-USGS Jan 15, 2025

lstanish-usgs Jan 15, 2025

ehinman Jan 2, 2025

ldecicco-USGS Jan 15, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025

ehinman Jan 2, 2025



		context("samples-data samples")
		test_that("samples-data samples working", {

	test_that("samples-data samples working", {
	test_that("samples-data activities working", {

	#' and the location number. Location identifiers should be separated with commas,
	#' and the location number, separated by a dash (-). Location identifiers should be separated with commas,


		## Return data types

		There are 2 arguments that dictate what kind of data is returned: dataType and dataProfile. The "dataType" argument define what kind of data comes back, and the "dataProfile" define what columns from that type come back.

Samples data #741

Are you sure you want to change the base?

Samples data #741

Conversation

ldecicco-USGS commented Dec 21, 2024

ehinman left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehinman left a comment •

edited

Loading