support for paginated endpoints

[ahl]

We should have first-class support for paginated endpoints, i.e. endpoints that enumerate lists of items and return a page at a time of content. We want to make it easy for developers to write these paginated endpoints without too much additional boiler plate.

In addition, it would be nice to have a generic manifestation of these paginated endpoints in the openapi doc. While OpenAPI doesn't have first-class support for pagination, we can potentially provide hints in the extensions (e.g. x-dropshot-paginated) that we could use in reference doc generation as well as client library code gen (both would require modifications to existing tools).

[davepacheco]

Adding a bunch of my own notes on this.

• The endpoints that do this today all return an HttpResponseOkObjectList, which is intended to encapsulate this sort of thing. It might need more context to fully do that (e.g., how many items are in the paginated set, how to construct the marker, etc.)
• The endpoints that do this today all consume a Query<PaginationParams<...>> and the pagination field is usually ApiName. (One paginates by uuid today.) This too is intended to encapsulate the input side of this use case.
• Internally in oxide-api-prototype, these endpoints are implemented using a common set of function (collection_page() and related functions).

The net result is that a lot of this is factored to be first-class already, though it maybe isn't quite there.

On my TODO list around this includes:

• (probably) swap NDJSON for an actual JSON object with fields for the marker, total_size, etc.
• if we can make this more first-class, it would be good to enforce limits on page size from a central place
• should the marker be an opaque token? should it be encrypted? (Google does do this.) Adam and I talked about having a v1 that's not encrypted. It would be pretty easy to re-do this provided that we put some recognizable version number in the representation (e.g., prepend "v1:" prior to base64-encoding).

[davepacheco]

See #14.

By the way, I read the Google example too quickly -- they only encrypt when there's sensitive information in the token.