Support efficient way of retrieving results for multiple path queries

[iilyak]

Given an example JSON I would like to be able to take arbitrary structure and extract multiple paths to return results as a flat dictionary. It can be done outside the rsonpath by calling it multiple times, but it would be really inefficient. It would be way better if we would be able to issue API call and collect the results in one go without traversing the data multiple times.

{
  "person": {
    "name": "John",
    "surname": "Doe",
    "phoneNumbers": [
      {
        "type": "Home",
        "number": "111-222-333"
      },
      {
        "type": "Work",
        "number": "123-456-789"
      }
    ]
  }
}

This is an example of the result

{
   "full_name": "John Doe" // concatenation of two fields
   "work_phone": "123-456-789"
}

The specification of the query could look like (I use JSON for demonstration purposes)

{
    "full_name": "rsonpath-query-to-calculate-full-name",
    "work_phone": "rsonpath-query-to-extract-work-phone"
}

[V0ldek]

Hey, thanks for interest in rsonpath!

Is this a request about the Rust library or are we talking about the rq CLI?

[iilyak]

Hey, thanks for interest in rsonpath!

Is this a request about the Rust library or are we talking about the rq CLI?

Rust library

[iilyak]

Some background. I wanted to contribute a feature into https://github.com/typst/typst which would take a data file (json/yaml) and given a query it would extract arbitrary fields and combinations of them. Then I would use typst language to generate a table and write it into pdf. The problem is that json file would be quite big (hundred of megabytes) but the result of each sub-query expected to be small. To avoid multiple scans (I need to extract about 20-30 different nested fields) I need something like either:

1. ability to run multiple queries and collect the results in the accumulator
2. iterator over json which would be checking matches against multiple queries
3. fold with engine which matches on multiple queries with accumulator

The #1 fits the bill perfectly. The #2 and #3 is kind of similar and essentially the same idea just implemented differently.

[V0ldek]

I wanted to contribute a feature into https://github.com/typst/typst

uu, helping with typst development would be exciting since I use it myself :)

The problem is that json file would be quite big (hundred of megabytes) but the result of each sub-query expected to be small. To avoid multiple scans (I need to extract about 20-30 different nested fields)

Okay, that's neat since it means we can very concretely test our performance against actual usage. I'd start by actually verifying how long running your queries separately takes. Can you tell me how to obtain this hundreds-of-megs JSON file and what queries you will run on it?

Modifying the engine to run multiple queries at the same time into multiple sinks would be feasible (at least as a prototype), but before going for a full design we'd need to measure how that compares against the baseline. There's also the immediate "brutal" solution to the problem, namely running your 30 queries on 30 threads, which has a real chance to be faster than any multi-query single-threaded solution. I'd be happy to experiment with a real JSON and queries :)

The specification of the query could look like (...)

Your idea of having a kind of an rsonpath specification on how to calculate a full result sounds neat, however it'd require some bespoke expression language to express stuff like "concatenate strings", so that's a huge design space. I think this particular issue though is solvable via JSONPath only, in your example the query:

$['person']['name', 'surname']

would give you ["John", "Doe"] and you can concatenate that after running the query. The spec guarantees the order. The issue is ofc that rsonpath doesn't support multiple selectors per segment yet, but that's something that's on the roadmap to support full JSONPath anyway. For now just splitting that into two queries would be a workaround.

Extracting the work phone is also problematic since it requires a filter, but that's something we are actively working on to solve (#156).