Pagefind as general faceting search engine

[stereobooster]

Backstory

I'm looking for client-side faceting search engine. See my article. I found 3 solutions: Tanstack table, Orama, Itemsjs.

Then I implemented my own search engine: facets

Then I realized that pagefind also capable of doing faceted search. (I missed it on my initial search). So I tried to use pagefind.

Repo: https://github.com/stereobooster/pagefind-instantsearch
Demo: https://pagefind-instantsearch.netlify.app/search/

But there are many obstacles in order to use it as general solution:

• doesn't support numbers for sorting
• doesn't support min, max for numeric facets
• doesn't support filtering by numeric facets (price > 50)

The question

Is this use case (general faceted search engine) something you would consider for the future?

[bglw]

Very cool demo! Thanks for sharing.

The question

Is this use case (general faceted search engine) something you would consider for the future?

Definitely. It isn't something that has had a lot of thought put in (yet), but I am keen to do some refinement on this use-case.

The obstacles

doesn't support numbers for sorting

Hmm, I'm interested to hear what you're experiencing here. Pagefind does implicitly support numbers for sorting, in that the data-pagefind-sort attribute can be given numbers, and Pagefind will endeavor to sort these numerically if possible.

Let me know if I'm missing the point of this somewhat!

doesn't support min, max for numeric facets
doesn't support filtering by numeric facets (price > 50)

Correct (currently). Some of Pagefind's limitations derive from the low-bandwidth constraint that everything is built under, and this is a great example of that. Implementing a fixed price > 50 filter would require knowing the concrete price of every result when querying, which Pagefind doesn't yet have (this data is loaded only for presentation).

Instead, sorting is effectively pre-calculated. So for a given set of results 0, 1, 2, 3, 4, 5, 6, 7, we also store that these results are 3, 4, 2, 7, 0, 1, 5, 6 when ordered by price. Since we only store this re-ordered set of indices, the bandwidth hit is minimal, and we can use the data at query-time.

The downside is that we don't know concretely what price 2 is in the example above — only that it is more expensive than 4 and less expensive than 7.

For most use-cases this is a good balance, so I'm keen to keep it as the default, but I'm open to exposing better information for faceted search implementations to use. Data like filters are already loaded as separate index chunks on demand, so it would be reasonable to output optional files containing more sorting data that can be loaded for applications that require them.

Depending on the size of the datasets you're wanting to target, we can also look at bandwidth reduction within these files — for example they only need to index various price points as indices, such that a lookup can determine that the $50 price point occurs at index 476 in the list of indices sorted by price, which querying can use to cull those results.

---

Anyway, that's a lot of my current thoughts on it 🙂 Let me know if this is something you're imminently wanting to pursue, and I can start looking at the API design we might need to support it well.

---

As an aside, these use-cases are also something I'm hoping to support better through the still-in-flux Modular UI, so that Pagefind can provide both the engine and the search frontend.

[bglw]

Gotcha. If you pass in:

sort: {
  price: item.price.toString()
}

Pagefind will later sort price numerically if all values can be parsed numerically, or alphabetically if not.

I'll make a note to accept numbers here — but it will still alias them to strings to integrate with the rest of the system.

[bglw]

The sorting tests here might help illustrate this: https://github.com/CloudCannon/pagefind/blob/main/pagefind/features/sorting.feature

Floats and integers will be parsed numerically and sorted 🙂

[stereobooster]

Indeed. I thought it doesn't work that way. Just updated the demo. Thanks

[stereobooster]

Let me know if this is something you're imminently wanting to pursue, and I can start looking at the API design we might need to support it well.

It's hard to tell what is a priority in this case. Facets and ItemsJS can serve as an example of API I'm looking for:

• it's possible to filter by facets without text search. E.g. even if there is no search query it still returns results (I think this would be the most important in my use case)
  • Blank search #546 (comment)
• By default pagefind returns all results. I imagine it loads all results in "WASM memory", but then also makes a copy of all results in main browser memory. I wonder if pagination would make sense in this case e.g. all results may still load in "WASM memory", but in main memory there would be only one page of results at a time
• Numeric facets:
  • for numeric facets I need min, max per facet. This required for slider (price in the demo)
  • and filtering by numeric facet, like price >= 50 and price <= 200
• Search for facets and sorting by frequency
  • Sometime facet list itself can be huge, so system needs a way to sort facets by frequency (most frequent on top) and cut off number of facets by, for example, 10
    • Sorting filter values #568
  • And in order to search for more facets there is a searchrbox - brands in the demo

But this is what I'm looking for. I do understand that you may have absolutely different use cases in mind. My wish-list is based on what typical faceted search does. For example, demos of other solutions:

• algolia + instantsearch
• facets + instantsearch

Let's discuss what is possible and what is not

[stereobooster]

Small update I did bunch of hacks and it kind of works, but there is a lot of not optimal code.