Add support for parsing content URLs for #204

[elipousson]

This PR adds preliminary support for the ArcGIS Portal Directory REST API by allowing arc_open() to work with item URLs and group URLs. I'd like to further extend this to also support user profile URLs and app URLs. These URLs are parsed to extract the item ID or group ID so this could be updated to allow a user to provide the item ID alone instead of an item URL.

This is a draft that I'm sharing early for feedback - I'll finish up the checklist below when the structure and approach look good. It is also possible some of the functionality for this PR related to URL handling should live in arcgisutils.

Checklist

• update NEWS.md
• documentation updated with devtools::document()
• devtools::check() passes locally

Changes

• Add utility functions to:
  • Detect when a URL is an item, group, user, or search URL: is_arc_content_url(),
  • Extract the url "type": arc_content_url_type()
  • Parse the ID value from the item or group URL and build a Portal API url arc_url_item()
• Modify arc_open() to support item and group URL inputs and, as needed, return a "FeatureService" or "MapService" class object
• Add examples demonstrating this new functionality

Issues that this closes

This PR makes progress on these two issues:

• Support providing IDs only #110
• Support for Portal Directory API aka Sharing API #204

Follow up tasks

• Allow functions to work with enterprise version of the Portal Directory REST API (not just the AGOL version)
• Update tests
• Review/update documentation

[JosiahParry]

This resolves to a map server:
https://services.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer

arc_open("[https://services.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer](vscode-file://vscode-app/Applications/Positron.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html#)")
#> <MapServer <19 layers, 0 tables>>
#> CRS: 3857
#> Capabilities: Map,Tilemap,Query,Data
#>  0: World Imagery (esriGeometryPolygon)
#>  1: Low Resolution 15m Imagery (esriGeometryPolygon)
#>  2: High Resolution 60cm Imagery (esriGeometryPolygon)
#>  3: High Resolution 30cm Imagery (esriGeometryPolygon)
#>  4: Citations (esriGeometryPolygon)
#>  5: 1.9cm Resolution Metadata (esriGeometryPolygon)
#>  6: 3.7cm Resolution Metadata (esriGeometryPolygon)
#>  7: 7.5cm Resolution Metadata (esriGeometryPolygon)
#>  8: 15cm Resolution Metadata (esriGeometryPolygon)
#>  9: 30cm Resolution Metadata (esriGeometryPolygon)
#>  10: 60cm Resolution Metadata (esriGeometryPolygon)
#>  11: 1.2m Resolution Metadata (esriGeometryPolygon)
#>  12: 2.4m Resolution Metadata (esriGeometryPolygon)
#>  13: 4.8m Resolution Metadata (esriGeometryPolygon)
#>  14: 9.6m Resolution Metadata (esriGeometryPolygon)
#>  15: 19m Resolution Metadata (esriGeometryPolygon)
#>  16: 38m Resolution Metadata (esriGeometryPolygon)
#>  17: 75m Resolution Metadata (esriGeometryPolygon)
#>  18: 150m Resolution Metadata (esriGeometryPolygon)

[JosiahParry]

Looks like a good start! Would you mind putting links to the API ref doc that you're referring to inside of the function definition so others can refer to it later?

In this code you're hard coding arcgis.com please use arc_host() so it respects the ARCGIS_HOST environment variable to support private portals or enterprise.

There's this function str_extract() which is from a package that does not have a compatible license so we gotta nix it. I think gsub() will work in its place.

The Feature Service and Map Service thing is and has been an oddity since the beginning. A feature service is any vector data hosted online. So a table is feature service, a polygon layer is a feature service, and so is a point. The distinction of FeatureServer and FeatureLayer is quite helpful for envisioning this hierarchical differentiation

[JosiahParry]

I'm a bit hesitant on this new arc_sharing() function.

Can you perhaps describe specifically what you're after and perhaps we can design something together? It feels like this PR is definitely very large and perhaps some of it belongs in arcgisutils.

[elipousson]

I kind of figured the additional changes might hit your limit. I think there are two main options that meet my needs:

• Allow arc_open() to parse and use item URLs that resolve to an existing supported type, e.g. Table, FeatureLayer, FeatureServer but not allow URLs for users, groups, or non-data items (e.g. PDFs, code samples, applications, etc.). In this case, passing an item URL or the URL for the related FeatureService would work identically. If this is the preferred approach, I'd like to turn arc_sharing() (or some equivalent) into a standalone utility function for working with the Sharing API (either w/in arcgislayers or w/in arcgisutils).
• Allow arc_open() to handle any content URLs and return the metadata. In this case, it probably makes sense to either document arc_sharing() separately or keep arc_sharing() as an internal function.

The URL parsing and building functions could be moved to arcgisutils. I'd also like to see arcgisutils::fetch_layer_metadata() extended to allow additional query arguments (and possibly get a new name since it gets metadata from anything - not just layers).

Does that make sense? I mostly want easy convenient access to the Sharing API in ways that make it easy to get information on items but I also think the convenience of providing an item URL instead of a FeatureService URL may be helpful for some users (see this comment for an example of how some users may be interested in using arc_open).

You may recall that I have a branch for arcgisutils where I added some URL validation functions back in the spring: https://github.com/elipousson/arcgisutils/tree/R-ArcGIS-main If any of that is still relevant, I could open a new PR over there with a more limited set of URL parsing, validating, and building functions.

[JosiahParry]

I think I'm starting follow but rather than talking about approaches, can we try and describe the objective?

From what I can tell there are two primary objectives:

• access an item based on its web server ui URL—e.g. https://analysis-1.maps.arcgis.com/home/item.html?id=9df5e769bfe8412b8de36a2e618c7672&sublayer=0
• get metadata for non-data-service items

The second one can be accomplished fairly straight forward using something like

set_arc_token(auth_user())

# https://developers.arcgis.com/rest/users-groups-and-items/item/
item_id <- "adb97a13b8b74f79b2f90463b34e8b78"

arc_base_req(arc_host(), query = c("f" = "json"), token = arc_token()) |> 
  req_url_path_append(c("sharing", "content", "items", item_id)) |> 
  req_perform() |> 
  resp_body_string() |> 
  yyjsonr::read_json_str()

The content of which can be fetched with

arc_base_req(arc_host(), query = c("f" = "json"), token = arc_token()) |> 
  req_url_path_append(c("sharing", "content", "items", item_id, "data")) |> 
  req_perform() 

Additionally for groups we can have:

group_id <- "asdfasfsafasfasdfdsa"

# https://developers.arcgis.com/rest/users-groups-and-items/group/
arc_base_req(arc_host(), query = c("f" = "json"), token = arc_token()) |> 
  req_url_path_append(c("sharing", "community", "groups", group_id)) |> 
  req_perform() |> 
  resp_body_string() |> 
  yyjsonr::read_json_str()

I would prefer that arc_open() is used to open any portal items. Groups and search queries feel semantically different. I want to avoid doing too much automagically. There's a balance that needs to be struck.

I think the most effective and also simplest thing that we can do now is make a utility function in arcgisutils that can handle web ui URLs that you're after.

This can return a list with the structure.

list(
    resource = "item|group|user|folder|search", 
    params = list(id = "", q = "", ...)
)

This structure can act as the backbone of everything else. The resource type can be matched on to identify the appropriate follow up function.