Optimize search feature

[thangisme]

Hi.
I'm trying to build a knowledge sharing site. I'm testing with about 4000 pages and my machine would crash instantly if I try to search even it's a really fast one. Is there anyway to optimize search speed or config alternative search engine?

[remysheppard]

Are you saying that you're running Hextra, building out 4,000 pages, trying to search, and it is in fact crashing?

Or are you saying you assume it would crash?

Also, an application of that size, you might actually be better off with a database solution than a static site.

[thangisme]

Are you saying that you're running Hextra, building out 4,000 pages, trying to search, and it is in fact crashing?

Or are you saying you assume it would crash?

Also, an application of that size, you might actually be better off with a database solution than a static site.

Now that I try again, it in fact not crash but really slow on my laptop. And on my phone, it just freeze the tab.

[thangisme]

And it's only slow on first search once the page load.

[thangisme]

It seems to load faster on chromium browser.
If you want, I can provide you the dev site link in private

[imfing]

It would be nice if you could share your repository in public, so I could figure out what's really going on.

I think 4000 pages definitely gonna overload the local search feature in Hextra.
Hextra pre-computes the search index file and when you open the search, the website load it on-the-fly. 4000 pages would result in very large index file including all the contents, thus the browsers or the search library just couldn't handle it.

In this case, I think integrating with third-party cloud offerings such as Algolia or Google Custom Search might be a better choice.

[thangisme]

It would be nice if you could share your repository in public, so I could figure out what's really going on.

I think 4000 pages definitely gonna overload the local search feature in Hextra. Hextra pre-computes the search index file and when you open the search, the website load it on-the-fly. 4000 pages would result in very large index file including all the contents, thus the browsers or the search library just couldn't handle it.

In this case, I think integrating with third-party cloud offerings such as Algolia or Google Custom Search might be a better choice.

I'm not gonna expose the repo in public.
But if you're up to implementing a alternative search engine. I might suggest Pagefind as I tried it locally and it's definitely much faster than the current one

[imfing]

It makes sense to only include titles when there are thousands pages. I can provide a flag for this in the future.

Not sure if this is the right way but I modify search-data.json as follow ($data to $pageTitle)

I doubt this would work correctly as we may also need to modify the search js script.

[thangisme]

Do you have plan to implement title only search in near future? I just tested several theme using flexsearch and they're also laggy

[imfing]

Do you have plan to implement title only search in near future? I just tested several theme using flexsearch and they're also laggy

By "laggy", do you mean using title only search or using full text search?

[imfing]

@thangisme Can you try the following to include only the title without the page data:

diff --git a/assets/json/search-data.json b/assets/json/search-data.json
index 16d3ad7..229e668 100644
--- a/assets/json/search-data.json
+++ b/assets/json/search-data.json
@@ -7,7 +7,7 @@
 {{- range $index, $page := $pages -}}
   {{- $pageTitle := $page.LinkTitle | default $page.File.BaseFileName -}}
   {{- $pageLink := $page.RelPermalink -}}
-  {{- $data := partial "utils/fragments" $page -}}
+  {{- $data := dict "" "" -}}
   {{- $output = $output | merge (dict $pageLink (dict "title" $pageTitle "data" $data)) -}}
 {{- end -}}

just a one-line change to override the assets/json/search-data.json as shown above, which should produce a much smaller search index json file.

[thangisme]

That did it. Thaank you!

[imfing]

FYI, I added configuration options different index types: content | summary | heading | title in #145