Improved search results

[iansinnott]

Search results could be improved. Some thoughts:

• allow infinite scroll or at least some level of customization of how many results are returned
  • the idea is generally that if you get too many results you should keep typing to narrow it down, but maybe you don't remember any more terms to narrow on
• allow limiting search to a specific site / url prefix (say, github.com)
• allow grouping by site, so that noisy sites are collapsed into a single list item unless expanded

Suggestions welcome!

[iansinnott]

Initially inspired by:

https://discord.com/channels/892246295585120297/892246295585120300/989964716967362591

It might be nice to allow summarizing by site - maybe show a list of sites with reveal triangles, and clicking the triangle shows all the results for that site.

[nfcampos]

Two more suggestions on this:

• more than grouping, for each exact URL (not domain) there should be only 1 entry, currently it looks like this, notice the same URL 4 times. I think for ideal results the de-duping should be done at ingestion time / prior to FTS index building?
• Related, consider a "frecency" ranking to combine with search query relevance (bm25 from sqlite?) ie. all else equal, a URL that was visited more often or more recently should rank above

[iansinnott]

ah yeah good point about the duplicate URLs. need to address that

[iansinnott]

#11

[nfcampos]

One more suggestion,

• downrank (but don't remove from results completely) homepages, ie. domain + / as imagine eg a blog, where I visited the homepage and a specific article linked from it. I'm almost certainly using BP to find the article, not its mention on the homepage. I think this should be generally true of uses of BP

[iansinnott]

interesting point. I'm thinking in terms of how to sort items that all have an exact match. do you think a rule like "longest URL first" would be helpful or as you said just downranking root paths?

There is already some logic to downrank search engines so this could be added in a similar fashion.

[nfcampos]

yea adding url length as another sorting fact sounds good. another consideration is an example like this

1. I go here https://daringfireball.net
2. I click into one of the posts https://daringfireball.net/linked/2022/06/23/flying-meat-twenty-years
3. I click into the linked post https://shapeof.com/archives/2022/6/twenty_years_of_flying_meat.html
4. I click into the homepage of what it is about https://flyingmeat.com

Then if I search for say "acorn" (a word found in all 4 sites) I'd want the ranking to probably be first 4 (then 3 then 2 then 1)

I think this is solved by making sure that the "longest path first" is only one more factor, and relevance can still trump it, as eg in this example the word acorn shows up many more times on 4, even though its a short url